Let me start first of all by saying that I apologize for the downtime we have experienced today.
I am sure that many of you have questions about what went wrong. As many of you know, early this morning, hurricane Wilma made landfall over Florida, making a beeline for the data center in Boca Raton. As many of you may remember, our data center is located in the former IBM World Headquarters which is designed to withstand a direct hit from a category 5 hurricane. During the barrage of last year’s storms across Florida we remained fully operational.
Early this morning when Wilma made landfall the data center made the switch to generator power, prior to the city grid going offline. These generators had been tested on Thursday the previous week and appeared to be fine. Shortly after throwing the switch on the generators this morning, things began to go wrong. The generator is actually a cluster of smaller generators (10MW) all housed in a separate building which is also capable of withstanding a direct hit from a category 5 storm in its own right. The generator utilizes 2 separate tanks of fuel, one large tank which feeds a smaller tank which runs the generators. This morning the system that feeds the fuel from the large tank to the small failed.
The estimated time of completion on a fix was just a few hours after they identified the problem point at the generator level. At this point, the entire data center was offline (roughly 19,000 servers in the building) before UPS power was completely exhausted, allowing the servers to be shut down properly. Verio engineers and facility managers of the building set to work on restoring generator power while Florida Power and Light was attempting to restore utility power to the area as soon as the winds died down as the storm moved across Florida. The reasons for the failure, and the reason that this was not something that could be fixed in a much more timely manner is as of yet unknown.
Please be aware that this has not been hardware issue. No hardware failure is apparent and all customer data is expected to be fully operational as soon as power is restored. We have explored alternative data center options throughout the day, however making a rash decision in this regard is undesirable. Not to mention that all of our customer site data would still be on the server in Boca. Our primary goal in this matter is to get all of our customer sites back online and be fully operational as quickly as possible. As of 12:30 AM Tuesday morning, Verio estimates that approximately 70% of the backup power system is functional and are working around the clock to bring the system up to an operational point. This is estimated to be before the start of business this Tuesday morning.
I would like to conclude with a heart felt thanks for your patience and understanding as we try to work through these issues. We are very much aware how important your web presence is to your business and how difficult it is to be without a working website and functional email. I have also been made aware of customer communication issues during a complete outage as we have experienced today and will be working to put in place a more robust system for the future, should the need arise. Once the system is back up and running, if you have questions or concerns, please create a support ticket at our online helpdesk for the fastest response.
Thank you again for your time and patience.
Back on the air...
Posted 25 October 2005 - 04:55 AM
Posted 25 October 2005 - 06:10 AM
But, Andrax is back!
Posted 25 October 2005 - 06:12 AM
01001001 00100000 01001100 01101111 01110110 01100101 00100000 01010000 01101001 01100101
Posted 25 October 2005 - 06:28 AM
Posted 25 October 2005 - 06:31 AM
Posted 25 October 2005 - 07:04 AM
Yesterday I was searching for JC's email address. I was going to yell at him because he had obviously dropped and broken his mouse this time, therefore taking down the boards!
Edited by Mofo Jones, 25 October 2005 - 07:07 AM.
Posted 25 October 2005 - 07:29 AM
Posted 25 October 2005 - 07:35 AM
The old "wind knocked out the power" excuse again! You damn Floridians
Posted 25 October 2005 - 08:24 AM
I guessed it had to be something to do with the hurricane. It didn't stop me from checking to see if things were back up every few hours.
I didn't think about the hurricane, but I was checking the boards every couple of hours yesterday too.
Good to be back
Posted 26 October 2005 - 03:41 AM
redundancy through duplication
Posted 26 October 2005 - 07:03 AM
Posted 26 October 2005 - 07:25 AM
Brigadier: Naturally enough, the only country that could be trusted with such a role was Great Britain.
The Doctor: Naturally. I mean, the rest were all foreigners.
Posted 26 October 2005 - 07:48 AM
Posted 26 October 2005 - 09:02 AM
Were the boards down again most of yesterday (Tuesday)?
Thats what I get for posting before reading the other thread titles.