Anatomy of an Outage
Written by Douglas Hanna on August 10, 2009 – 4:02 pm -Wednesday, August 5, 2009 started out as a normal day at HostGator’s Houston headquarters. Around 4:00 PM CT, a major power surge that occurred as the result of a transformer near our office blowing up made the day anything but ordinary.
Lights flickered, battery backups beeped, fire alarms went off, and Internet signals all died down almost immediately. People began to wait for the building’s $200,000 hurricane-ready generator to start up, but it didn’t.
In the mean time, one of the three major “legs” of power that feeds the building with the power it needs to function was out because of exploded transformer. The building was underpowered and the higher voltage motors and equipment started burning out from the heat and stress of running without the adequate amounts of power. Expensive equipment continued to get damaged.
A compressor on the air conditioning burnt out (cost: $35,000), air handlers got destroyed (cost: $5,000), an elevator motor got fried (cost: $10,000) and lots of other equipment in the building’s mechanical room still isn’t working correctly (cost: unknown). The total cost of the damages is expected to be upwards of $60,000.
As the building’s systems started to go down and the people in charge of HostGator’s office began calling in electricians, power companies, and repairmen, the rest of the management team began going into what we refer to internally as “hurricane mode.”
- Twitter updates started to go out informing customers of a power problem in the building and possible service delays.
- Employees were rallied and were sent to the other employees’ homes.
- Our phone number was redirected (our VOIP system is housed in our office) and the message on our phone system was updated to inform customers of the outage.
- Our support site was updated with an emergency notice.
- A forum post was made with additional details.
As the makeshift offices were being setup in our managers’ homes, chats were being taken, servers were being monitored, and updates were being provided. Within an hour of the surge, HostGator’s support operations were almost fully functional, albeit delayed (with the exception of phone support).
By 11:30 PM, employees were starting to work at the office again. The phones were turned on shortly afterwards and average email response times went back down to 45 minutes or less.
Much of this expensive and inconvenient damage would have been prevented had the building’s generator worked as planned. If it did, the building would have only lost power for a minute or so instead of multiple hours. The cause was the generator maintenance done less than a week before (by an outsourced company) was done improperly. The company put the wrong fuel filter on the generator, which caused the generator to immediately fail on start up.
The outage could have obviously been much worse. No customer servers or accounts were affected in any way (we don’t house any customer servers in our office building) and we were able to get back up and running relatively quickly.
Regardless of the relative severity of the event, though, HostGator did learn a lot.
- Most notably, the fact the immediate communication is essential was reaffirmed. We first learned about the importance of immediate communication during a datacenter outage at The Planet. In this situation, a Twitter update went out less than 15 minutes after the power surge occurred. Updates continued to be provided across Twitter, the forums, and our support site until the situation was completely resolved. We were even lucky enough to get comments from customers praising us for our handling of the situation.
- We also learned that it’s critical to have systems tested and maintained by companies we know are getting the job done properly. We are obviously looking into a new generator maintenance company and looking at our other vendors to ensure they’re prepared to deal with issues if they occur.
During the entire occurrence, our customers were patient and understanding and we sincerely appreciate that. Stanley Marcus of Neiman Marcus fame is credited with saying “The road to success is paved with well handled mistakes” and we couldn’t agree more.
Things happen (the web hosting business and the act of running a business are never dull) and Wednesday’s events were just one of the many examples of things that no one could have ever predicted happening.
Click on the images below to see a larger version with a caption.
Posted in Customer Service, Random | 23 Comments »
Giving back with completely free web hosting.
Written by Douglas Hanna on September 29, 2008 – 1:13 pm -HostGator has a history and tradition of giving back. In the last year, we’ve donated over $125,000 to charity, actively worked towards offsetting and reversing our environmental impact, and given free or
discounted web hosting to a countless number of web sites.
Today, we’re moving forward with our commitment to the communities and the people we serve with the launch of the HostGator Technology Grant.
Recipients of the HostGator Technology Grant will receive one year of completely free web hosting with our Swamp Plan, which offers 1,000 GB of space and unlimited bandwidth. If the Swamp plan does not fit the needs of the particular recipient, they can opt to take the grant as a $180 service credit to any other HostGator product or service. After the one year of free service, grant recipients can re-apply and if accepted again, continue to use HostGator web hosting completely free.
The HostGator Technology Grant’s goal is simple and straightforward: provide free web hosting to organizations and projects that serve the greater good.
Obtaining legitimately free web hosting for nonprofits can be an arduous process. Organizations should be spending their time doing what they believe in and what they set out to do, not hunting for
affordable and quality web hosting.
The wide scope of the grant is intentional – we wanted it so all types of organizations that serve the greater good could take advantage of free HostGator web hosting. Whether your organization is nonprofit community center or a cutting edge open source project, we want to hear from you.
The application process for the HostGator Technology Grant is simple. There are just a few questions and it shouldn’t take an organization longer than 15 or 20 minutes to complete the short application. Click Here to APPLY
Once you’re all setup, then you can do your actual job and get back to doing what you love. We want to make the web hosting experience less troublesome and more convenient for everyone, including those who are seeking to better society through their organizations and projects.
If you have any questions about HostGator’s Technology Grant program, please go to http://www.hostgator.com/nonprofit-web-hosting.shtml or email techgrant@hostgator.com. We’re more than happy to help.
Posted in Gator Goodness, News Bites, Promotions, Random, Web Hosting News, World Gator | 7 Comments »
Blog Awards And Your Votes
Written by Justin G on January 10, 2008 – 2:03 pm -
I’m sure if you read more than several blogs you may have seen the logo thats in the top right corner of this blog right now. The bloggers choice awards are given for the top blogs of multiple categories. The bloggers choice awards can give you some nice exposure as a blogger as well as give you something to gloat about if you win. Its pretty much the Oscar award of blogging so to speak.
Besides the Bloggers Choice awards, there are other awards for bloggers as well. I thought I might put together a few of the other prominent ones for those of you who not only read the HostGator blog, but blog yourself as well.
Since I have started writing for the blog, I have gotten a chance to get to know some really cool people a bit better and read over others blogs that have great content. We always appreciate your support, and especially from you feed readers that are subscribed. I thought it would be good to get involved with the Bloggers Choice awards now that things have gotten on a roll, and so I have added a nomination link for it as I mentioned earlier in the top right corner.
If you enjoy reading this blog then feel free to vote for it. Also, tell me what blog you have that might be nominated for a particular subject. I would be more than happy to check it out and vote for yours as well. We have a lot of fun things coming up to post about, so stay tuned, and if you haven’t joined our feed I would encourage you to.
Posted in Random | No Comments »
Stuck In The Elevator
Written by Justin G on January 2, 2008 – 11:16 am -When Daniel and Julio (HostGator support) thought that it was going to be just an average day at work, they had no idea what was coming. After entering the Elevator of the HostGator Building on the first floor, and pushing the 4th floor button, to their surprise the elevator stopped halfway up. What do people do in a small elevator when its stuck? They call in late to work to start with, because if you get stuck in an elevator there is a chance you may be there for a long time. I think 2 hours was enough for them.
After enough standing around, they decide to sit down. Since it was the holidays and there was nobody really working on elevators, Julio and Daniel thought they might have to wait till the next day to get out, but they didn’t. Brent and Brian managed to prop open the doors to help them squeeze through to the next floor. Just before the rescue, Daniel managed to record their cramped experience with his phone. From the looks of it Julio is a little upset already, Daniel however just seems to be having fun with this whole thing. Check the video out that we put on Youtube below.
Posted in Random | 7 Comments »
500,000 Dollar Google USB Drive
Written by Justin G on December 11, 2007 – 7:17 pm -I’m sure you might be expecting a USB drive made of Gold with inlay diamonds sparkling away, but all that glitters isn’t gold, and as a matter of fact, you won’t find anything at all that is absolutely stunning. We merely want to show you what you could expect if you were to lay down about 500 big ones for Google advertising. We received the annual gift from Google and just wanted to show you what its all about.
The presentation was nice and Google does a good job of customizing their gifts while incorporating a touch of advertising. The video below shows us opening the gift, and you can see by our excitement that when you get something from Google, it’s special. This is the video of what we like to call the $500,000 Google USB Drive.
You choose where it goes
So if you watched the video you will notice we received a card for DonorsChoose.org, and it appears we have $100 dollars in credit to donate to a cause related to schooling. We went through the list of causes to donate to and had a hard time choosing something. We want to leave it open to you, the people that read this blog, to give us some feedback with your comments and tell us what cause you think would be the best.
Posted in Random | 21 Comments »
