Jump to content


Photo

Back on the air...


  • Please log in to reply
18 replies to this topic

#1 Andrax

Andrax

    Andrax.Net Admin

  • Site Admin
  • 9,376 posts

Posted 25 October 2005 - 04:55 AM

Well, I would imagine that some of you might have noticed that we were down for almost 24 hours. Here's why...

Let me start first of all by saying that I apologize for the downtime we have experienced today.

I am sure that many of you have questions about what went wrong. As many of you know, early this morning, hurricane Wilma made landfall over Florida, making a beeline for the data center in Boca Raton. As many of you may remember, our data center is located in the former IBM World Headquarters which is designed to withstand a direct hit from a category 5 hurricane. During the barrage of last year’s storms across Florida we remained fully operational.

Early this morning when Wilma made landfall the data center made the switch to generator power, prior to the city grid going offline. These generators had been tested on Thursday the previous week and appeared to be fine. Shortly after throwing the switch on the generators this morning, things began to go wrong. The generator is actually a cluster of smaller generators (10MW) all housed in a separate building which is also capable of withstanding a direct hit from a category 5 storm in its own right. The generator utilizes 2 separate tanks of fuel, one large tank which feeds a smaller tank which runs the generators. This morning the system that feeds the fuel from the large tank to the small failed.

The estimated time of completion on a fix was just a few hours after they identified the problem point at the generator level. At this point, the entire data center was offline (roughly 19,000 servers in the building) before UPS power was completely exhausted, allowing the servers to be shut down properly. Verio engineers and facility managers of the building set to work on restoring generator power while Florida Power and Light was attempting to restore utility power to the area as soon as the winds died down as the storm moved across Florida. The reasons for the failure, and the reason that this was not something that could be fixed in a much more timely manner is as of yet unknown.

Please be aware that this has not been hardware issue. No hardware failure is apparent and all customer data is expected to be fully operational as soon as power is restored. We have explored alternative data center options throughout the day, however making a rash decision in this regard is undesirable. Not to mention that all of our customer site data would still be on the server in Boca. Our primary goal in this matter is to get all of our customer sites back online and be fully operational as quickly as possible. As of 12:30 AM Tuesday morning, Verio estimates that approximately 70% of the backup power system is functional and are working around the clock to bring the system up to an operational point. This is estimated to be before the start of business this Tuesday morning.

I would like to conclude with a heart felt thanks for your patience and understanding as we try to work through these issues. We are very much aware how important your web presence is to your business and how difficult it is to be without a working website and functional email. I have also been made aware of customer communication issues during a complete outage as we have experienced today and will be working to put in place a more robust system for the future, should the need arise. Once the system is back up and running, if you have questions or concerns, please create a support ticket at our online helpdesk for the fastest response.

Thank you again for your time and patience.



#2 oVo

oVo

    Thought Mod. Wannabe

  • Valued Member
  • PipPipPipPipPipPipPipPipPipPip
  • 9,892 posts

Posted 25 October 2005 - 06:10 AM

Ah, Wilma. Should have known.


But, Andrax is back! :w00t:
Posted Image [font="Impact"]

#3 RogueTrooper

RogueTrooper

    Forum Demi-God

  • Valued Member
  • PipPipPipPipPipPipPipPip
  • 1,963 posts

Posted 25 October 2005 - 06:12 AM

The old "wind knocked out the power" excuse again! You damn Floridians ;)
Posted Image
01001001 00100000 01001100 01101111 01110110 01100101 00100000 01010000 01101001 01100101

#4 Andrax

Andrax

    Andrax.Net Admin

  • Site Admin
  • 9,376 posts

Posted 25 October 2005 - 06:13 AM

Heh, i'm not in Florida, but the server is! ;)

#5 oVo

oVo

    Thought Mod. Wannabe

  • Valued Member
  • PipPipPipPipPipPipPipPipPipPip
  • 9,892 posts

Posted 25 October 2005 - 06:28 AM

All this time I thought Andrax was trying to fix the gray page number indication boxes. :lol:
Posted Image [font="Impact"]

#6 Andrax

Andrax

    Andrax.Net Admin

  • Site Admin
  • 9,376 posts

Posted 25 October 2005 - 06:31 AM

heh, no. Thats pretty low on my list of priorities because it involves searching through the CSS of the skin line by line trying to figure out which variable references that. Even if i do find it, I can almost guarentee that its tied to something else so changing that color will screw up something else, hence my inclination to leave it the hell alone.

:devil:

#7 oVo

oVo

    Thought Mod. Wannabe

  • Valued Member
  • PipPipPipPipPipPipPipPipPipPip
  • 9,892 posts

Posted 25 October 2005 - 06:47 AM

Good man.
Posted Image [font="Impact"]

#8 Mofo Jones

Mofo Jones

    Forum God

  • Moderators
  • 12,268 posts

Posted 25 October 2005 - 07:04 AM

WIIIIIIIIIIIIIIIIIIIIIIIIILLLLLLLLLLLLLLMMMMMMMMMMMMMMAAAAAAAAAAAAAAAAAA!!!!!!

Yesterday I was searching for JC's email address. I was going to yell at him because he had obviously dropped and broken his mouse this time, therefore taking down the boards! B)

Edited by Mofo Jones, 25 October 2005 - 07:07 AM.

Posted Image

#9 Farfik

Farfik

    Forum God

  • Valued Member
  • PipPipPipPipPipPipPipPipPipPip
  • 3,583 posts

Posted 25 October 2005 - 07:29 AM

I guessed it had to be something to do with the hurricane. It didn't stop me from checking to see if things were back up every few hours.

#10 Johnny Cool

Johnny Cool

    Johnny Cool

  • Moderators
  • 15,863 posts

Posted 25 October 2005 - 07:35 AM

The old "wind knocked out the power" excuse again! You damn Floridians ;)


:lol: :lol:

#11 Paco DEmu

Paco DEmu

    Master Poster

  • Members
  • PipPipPipPipPipPipPip
  • 1,197 posts

Posted 25 October 2005 - 08:24 AM

I guessed it had to be something to do with the hurricane. It didn't stop me from checking to see if things were back up every few hours.


I didn't think about the hurricane, but I was checking the boards every couple of hours yesterday too.

Good to be back :)
Don't hate the pie, hate the crust.

#12 Woody Dudat

Woody Dudat

    Forum Supreme Being

  • Valued Member
  • PipPipPipPipPipPipPipPipPip
  • 2,603 posts

Posted 26 October 2005 - 03:41 AM

:unsure:
redundancy through duplication
redundancy through duplication

#13 oVo

oVo

    Thought Mod. Wannabe

  • Valued Member
  • PipPipPipPipPipPipPipPipPipPip
  • 9,892 posts

Posted 26 October 2005 - 03:53 AM

:unsure:



:huh:
Posted Image [font="Impact"]

#14 MrBastage

MrBastage

    Forum Demi-God

  • Valued Member
  • PipPipPipPipPipPipPipPip
  • 2,021 posts

Posted 26 October 2005 - 07:03 AM

Thanks for the note Andrax, I was wondering what happened. Looks like Fred needs to make sure Wilma stays in the house in Bedrock instead of letting her go out galavanting!

Posted Image


#15 Menelaus

Menelaus

    Mini-mene

  • Moderators
  • 13,841 posts

Posted 26 October 2005 - 07:25 AM

I wonder if lobsta is alright?
"It is the job of the judges to ensure that the government of the day does not exceed its powers, which is a permanent desire of all governments." Lord Donaldson of Lymington

Brigadier: Naturally enough, the only country that could be trusted with such a role was Great Britain.
The Doctor: Naturally. I mean, the rest were all foreigners.

#16 Farfik

Farfik

    Forum God

  • Valued Member
  • PipPipPipPipPipPipPipPipPipPip
  • 3,583 posts

Posted 26 October 2005 - 07:48 AM

Were the boards down again most of yesterday (Tuesday)?

#17 Paco DEmu

Paco DEmu

    Master Poster

  • Members
  • PipPipPipPipPipPipPip
  • 1,197 posts

Posted 26 October 2005 - 08:13 AM

Were the boards down again most of yesterday (Tuesday)?



yup
Don't hate the pie, hate the crust.

#18 Farfik

Farfik

    Forum God

  • Valued Member
  • PipPipPipPipPipPipPipPipPipPip
  • 3,583 posts

Posted 26 October 2005 - 09:02 AM

Were the boards down again most of yesterday (Tuesday)?



yup


Thats what I get for posting before reading the other thread titles. :pinch:

#19 oVo

oVo

    Thought Mod. Wannabe

  • Valued Member
  • PipPipPipPipPipPipPipPipPipPip
  • 9,892 posts

Posted 26 October 2005 - 09:29 AM

Saved me the trouble Farfik.
Posted Image [font="Impact"]