jump to navigation

Node 3 Should Be Working March 23, 2011

Posted by ismywebsite in general.
trackback

So I just received the following:

As you should be aware, all hosting accounts which were offline since [Node 3] went down on Thursday till Sunday, have now all been restored and should all be active. [You will now have to update the IPs manually.]

Now to clarify what happened – the DataCenter housing [Node 3] had a UPS battery failure which let off some smoke on Thursday morning. This activated the fire alarms which in turn automatically shut down all power to the facility as per design. After an hour power was restored via generators once the all clear was given by the fire service and later on that evening full power was transferred back to the mains with UPS protection enabled.

Due to the power issues, the PSU [(power supply unit)] and hard drives on [Node 3] failed. After testing the primary hard drive back in our office it was not even being recogni[z]ed at BIOS level. Hence we reverted to our bare metal backups on our external CDP [(I believe this refers to either Continuous Data Protection or Cisco Data Protocol, not sure which is in use.)] server. Restoring this data took over 24 hours, and because they were bare metal backups we needed to first mount the files/directories on another server to take meaningful backups from them due to the way in which [cP]anel backups are created. Once this was completed, the backups were restored on [the new server] along with the MySQL data.

Now for many of you, this technical information will be hard to follow and most likely will not even be relevant as your main concern is uptime. We appreciate this and sincerely apologi[z]e for this extended outage, but must point out to you that it was totally out of our control and that hardware can sometimes fail. As per design, we had the necessary backups to restore all data albeit after 72 hours of downtime.

[W]e will try our best to ensure this kind of situation does not occur again.

So, things should be back online shortly. I just need to update the DNS to reflect this server change.

Advertisements

Comments»

1. J,R,D, Ltd - March 23, 2011

Well at least they had backups and kept us posted.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: