HostGator Web Hosting Blog | Gator Crossing » Customer Service

Anatomy of an Outage

Douglas Hanna — Mon, 10 Aug 2009 21:02:50 +0000

Wednesday, August 5, 2009 started out as a normal day at HostGator’s Houston headquarters. Around 4:00 PM CT, a major power surge that occurred as the result of a transformer near our office blowing up made the day anything but ordinary.

Lights flickered, battery backups beeped, fire alarms went off, and Internet signals all died down almost immediately. People began to wait for the building’s $200,000 hurricane-ready generator to start up, but it didn’t.

In the mean time, one of the three major “legs” of power that feeds the building with the power it needs to function was out because of exploded transformer. The building was underpowered and the higher voltage motors and equipment started burning out from the heat and stress of running without the adequate amounts of power. Expensive equipment continued to get damaged.

A compressor on the air conditioning burnt out (cost: $35,000), air handlers got destroyed (cost: $5,000), an elevator motor got fried (cost: $10,000) and lots of other equipment in the building’s mechanical room still isn’t working correctly (cost: unknown). The total cost of the damages is expected to be upwards of $60,000.

As the building’s systems started to go down and the people in charge of HostGator’s office began calling in electricians, power companies, and repairmen, the rest of the management team began going into what we refer to internally as “hurricane mode.”

Twitter updates started to go out informing customers of a power problem in the building and possible service delays.
Employees were rallied and were sent to the other employees’ homes.
Our phone number was redirected (our VOIP system is housed in our office) and the message on our phone system was updated to inform customers of the outage.
Our support site was updated with an emergency notice.
A forum post was made with additional details.

As the makeshift offices were being setup in our managers’ homes, chats were being taken, servers were being monitored, and updates were being provided. Within an hour of the surge, HostGator’s support operations were almost fully functional, albeit delayed (with the exception of phone support).

By 11:30 PM, employees were starting to work at the office again. The phones were turned on shortly afterwards and average email response times went back down to 45 minutes or less.

Much of this expensive and inconvenient damage would have been prevented had the building’s generator worked as planned. If it did, the building would have only lost power for a minute or so instead of multiple hours. The cause was the generator maintenance done less than a week before (by an outsourced company) was done improperly. The company put the wrong fuel filter on the generator, which caused the generator to immediately fail on start up.

The outage could have obviously been much worse. No customer servers or accounts were affected in any way (we don’t house any customer servers in our office building) and we were able to get back up and running relatively quickly.

Regardless of the relative severity of the event, though, HostGator did learn a lot.

Most notably, the fact the immediate communication is essential was reaffirmed. We first learned about the importance of immediate communication during a datacenter outage at The Planet. In this situation, a Twitter update went out less than 15 minutes after the power surge occurred. Updates continued to be provided across Twitter, the forums, and our support site until the situation was completely resolved. We were even lucky enough to get comments from customers praising us for our handling of the situation.
We also learned that it’s critical to have systems tested and maintained by companies we know are getting the job done properly. We are obviously looking into a new generator maintenance company and looking at our other vendors to ensure they’re prepared to deal with issues if they occur.

During the entire occurrence, our customers were patient and understanding and we sincerely appreciate that. Stanley Marcus of Neiman Marcus fame is credited with saying “The road to success is paved with well handled mistakes” and we couldn’t agree more.

Things happen (the web hosting business and the act of running a business are never dull) and Wednesday’s events were just one of the many examples of things that no one could have ever predicted happening.

Click on the images below to see a larger version with a caption.

Talk to HostGator Live on July 29 at 8 PM CT

Douglas Hanna — Wed, 15 Jul 2009 22:39:59 +0000

Update: Thanks to all of those who participated! Our second Open Session was a success. A recording of the session is available on this page.

Due to the success of the first Open Session we hosted a couple of weeks ago, we’re hosting another one soon. This Open Session will be held on Wednesday, July 29 at 8 PM CT.

Just like last time, this Open Session will be a chance for both potential and existing HostGator customers to come into a live chat with other customers and HostGator employees and get tips, information, and have a chance to ask questions and get answers from the people who make the decisions at HostGator. Participants can either call in or just listen on their computers. There is also a text chat for people who would prefer to type their questions.

Some of the things we’ll be discussing include:

Tips and suggestions on how to get the most out of your HostGator account from senior system administrators
Discussion of HostGator’s upcoming Windows Shared Hosting launch, including questions and answers with the person in charge of Windows hosting here at HostGator
Questions and answers about anything related to HostGator, ranging from what our servers support to what we eat for lunch in the office
Discounts and/or free hosting to some participants who ask great questions.

Like last time, we’re planning on having our Deputy CTO, Chief Marketing Officer, and Customer Service Manager lead this Open Session. These three people are able to answer just about any question you can think of related to HostGator. There will also be senior support people available to answer more general questions and help out as needed.

Logistics:
TalkShoe worked well for us last time and we’ll be using it again for this Open Session. If you live outside of the US central timezone, check out this site for a list of corresponding times around the world. Right before 8 PM CT on Wednesday, head over to this page for instructions on how to join the session (it’s very simple).

Overview:
What: HostGator Open Session
When: Wednesday, July 29, 2009 at 8 PM CT
Where: HostGator on TalkShoe

Chat with HostGator this Thursday at 5 PM CT

Douglas Hanna — Wed, 01 Jul 2009 00:13:24 +0000

Update: This took place and was a big success. HostGator employees answered a ton of questions about backups, VPS solutions, Windows hosting, affiliates, and more. We appreciate people taking the time to come by and ask questions.

This upcoming Thursday (July 2, 2009) at 5 PM CT (Houston-time), HostGator is going to be hosting an Open Session where potential and existing customers alike can come chat with HostGator employees and each other.

The Open Session is going to be pretty informal, but for some context, some things we do want to do include:

Question and answer session with HostGator employees
Tips and suggestions on how to get the most out of your HostGator account and website from our best support people
Suggestions from customers about how HostGator can improve (customer service, Terms of Service, procedures and policies, etc.)

This is the first time HostGator has done something like this, but we think it will serve as a great opportunity to talk to our customers and hear their questions, opinions, and suggestions. If you have any particular items you’d like us to discuss or research, feel free to leave them in the comments.

Oh, and we’re also going to be giving out some free hosting during the session for both attending and participating.

Logistics:
We’re going to try hosting this session with TalkShoe, which allows us and other folks to call in using the phone and/or chat or listen online. If it works, we’ll continue to use TalkShoe. If not, we’ll explore other options. If you live outside of the US central timezone, check out this site for a list of corresponding times around the world. When Thursday at 5 PM CT rolls around, head over to this page for instructions on how to join the session (it’s very simple).

Overview:
What: HostGator Open Session
When: Thursday, July 2, 2009 at 5 PM CT
Where: HostGator on TalkShoe

All TPS reports require cover sheets.

Douglas Hanna — Fri, 03 Apr 2009 16:53:34 +0000

If you’ve ever seen Office Space, you most likely know that all TPS reports (that’s “Test Procedure Specification” for the uninformed) require cover sheets. The painfully ridiculous incident in the movie has become both a critical fixture of anti-corporate culture and a quintessential example of the eccentricities of bureaucracies of corporate America.

At HostGator, there are no TPS reports and cover sheets aren’t required to be put on anything. However, we do keep track of quite a few numbers and metrics and one of the metrics we do watch closely is chat volume.

The graph below is an average of the number of chats we completed per hour on any given day and at any given time throughout the month of March. The numbers are averaged, meaning that every Monday at 1 PM was averaged together to create a collective Monday at 1 PM metric. Our week-to-week variation is usually pretty negligible, so what you see on the graph is a fairly good representation of our chat volume for a given day. Note that the Y-axis starts at 30, meaning we never average less than 30 chats completed per hour.

(Click to see the full size version.)

The graph is a fairly normal distribution and makes it quite clear that we’re usually busiest right around noon central time on Mondays and Tuesdays. Our slowest times (on average) tend to be during the very early mornings on the weekends. 2 AM on Sunday is almost always dead and weekends in general tend to be fairly quiet when compared to the busy weekdays.

Keep in mind that all of these numbers are averages and they have to be taken as that. At the time of writing this post, things are actually slower than usual here. This morning was much busier than our typical Thursday morning and we only guess what tomorrow will be like. Our data indicates that it will be fairly busy for most of the day, but it could also end up being unusually quiet.

The hosting industry is an industry characterized by some amount of unpredictability. We do our best to staff our live chat and phone support departments according to the volume we’ve seen, but there are always going to be days when we’re overstaffed and days when we’re understaffed.

Unpredictability in a constantly changing industry like web hosting and at a constantly changing company like HostGator is a fact of life and much to the disappointment of the executives at Initech, not something that a cover sheet on a TPS report can mask.

Twittering All The Way

Douglas Hanna — Thu, 01 Jan 2009 02:59:06 +0000

I’ve known about Twitter for a relatively long time, but could never see myself using the service. I just couldn’t grasp the idea of saying something meaningful in 140 characters or less, but as I started working at HostGator and our social media outreach programs, I found myself thrown right into the Twittersphere (if that isn’t a word already, I just made it up).

A couple of months later, though, I’m happily Twittering from the web and from my ever trusty BlackBerry. A majority of our presence on Twitter is aimed at addressing customer concerns and feedback in what is essentially real time, but our presence on Twitter is also about reaching out to customers through a different medium and in a unique way. We already reach out to those who blog or post about us, so Twitter seemed like the next logical step.

So far, it’s been incredibly successful. We have over 275 followers on Twitter (the number increases daily) and I have personally reached out to and helped a lot of existing and potential customers through Twitter. We’ve also gotten some great ideas and some excellent feedback that we’ve taken seriously.

To get an idea for yourself, check out our favorites and see what people have been saying. Some excerpts:

LOVE LOVE HostGator! Have hosted at many places. Couldn’t dynamite me out of HG! – WordPressWizard

Thanks for awesome customer service, that’s how it’s done. Will remember when someone is asking for a hosting company. – KatjaPresnal

Your help today convinced me to sign up for 2 years of hosting. Thanks again! I’m uploading now at ~1400kbps. Amazing! – steelopus

I REALLY appreciate that you care about your customers, even on twitter though. Impressive! – BenTremblay

I appreciate your presence on Twitter. It’s more convenient for me to read your tweets than it is to go to your site. – keithgoode

Well, Madame or Sir HostGator, you sure know how to solidify a customer with service that spoils. Major thanks! – Rich_Arnold

Of course, our own Twitter stream can serve as another example. You’ll mostly see me reaching out to and responding to customers, but you’ll also see some other personal tidbits and web hosting related thoughts as well.

Here are some other things you can look forward to seeing on HostGator’s Twitter stream in the near future:

Survivor: HostGator Manager Editor. I am going camping for the first time with a group of my friends very soon and if my BlackBerry still works (which I am really hoping it will), I will be providing updates over Twitter.
More quotes and interesting thoughts. I tweeted that it was time for our Twitter followers to see my inner-nerd (because I obviously conceal my inner-nerd well by working for a web hosting company) and I think interesting quotes and thoughts from some equally interesting people is a good way to show it.
Web hosting tips. I’m going to begin posting some tips (technical and not) about how to get the most out of your web hosting account right on our Twitter account.

Check out our Twitter stream and if you like what you see, consider following us. And of course, if you have any suggestions for what we should do with our presence on Twitter, don’t hesitate to let us know.

Numbers That Matter

Douglas Hanna — Fri, 25 Jul 2008 16:04:03 +0000

Not everything that can be counted counts, and not everything that counts can be counted.
- Albert Einstein

I work with many companies of many sizes through my consulting business. When HostGator offered me the opportunity to conduct a customer satisfaction survey for them, I jumped at the chance. Even though customer satisfaction as a topic profoundly interests me, I have generally held the view that numbers themselves are painfully boring. I’ve never liked math and I have always felt that numbers are things that are best left to the experts who actually like numbers.

Over the last two weeks, though, I’ve been immersing myself in numbers – especially numbers relating to customer satisfaction. HostGator has sent out nearly ten thousand surveys to random customers and people who have completed live chats. The result? A lot of data (and a lot of numbers) on what HostGator’s customers think of the company and how its doing.

Fortunately, the results we gathered from all that data were neither shocking nor surprising. While there were no astonishing revelations, there was data that confirmed what HostGator thought based on gut feeling and there was data that showed everyone some interesting trends and correlations.

That data showed that just about 90% of HostGator customers gave a positive response when asked how likely they were to recommend HostGator to a friend or colleague. The average “would you recommend” rating was 8.36.
A similar percentage of HostGator customers felt the company’s employees were able to resolve their problems. The average “agent performance” answer was 8.55.
Roughly 83% of the customers surveyed through post-chat surveys responded saying that HostGator resolved their issues in a timely manner.
About 91% of HostGator’s customers rated the customer service ability of HostGator’s employees positively.
In every survey conducted with a 0 to 10 scale, 10 was the most popular choice.

Even though the ratings HostGator received were pretty good by essentially any standard, HostGator wants 90% or more of ratings at an exemplary, not a “pretty good” level. To HostGator’s management team and to me, exemplary means means that just about 100% of a company’s ratings would be 9’s and 10’s. Averages would then be in the mid 9’s instead of the mid 8’s.

The fact that HostGator was willing to conduct the surveys and then share the results shows a lot about the company.

In the business world, there is no shortage of companies that approach customer satisfaction with complete confidence that the results will be overwhelmingly positive. As the results come in, the executives at these companies quickly learn that ignorance is not bliss and does not provide for adequate protection against shocks to the ego.

These executives see painfully disappointing results and even more disappointing feedback from customers, all of which indicate that the real world is very different from what they had assumed.

HostGator has done just the opposite. Not only have they asked me to share the results of their surveys, but they have already started making changes and working on improvements. HostGator’s management team is not conducting these surveys just to see where they are and leave it at that – they are conducting these surveys to gather the information necessary to make changes and to improve