LoGro was unavailable for an hour last night

This morning I wrote to Nami Teramoto, one of the staffers at TigerTech:

It would have been helpful to get a heads-up that the sites were being moved last night and that the domain aliases might be an issue. I was in panic mode for an hour, tweeting and blogging and answering emails.

All I could see on the TT blog/Twitter account was that another Denial of Service Attack was happening which didn’t make sense for my problem since wigleyandassociates.com was up/working.

Nami replied:

Griff,

Thank you for writing! We are very sorry we weren’t able to give you any prewarning on the move. Unfortunately, due to the DDoS attack at the old data center, we were forced to do the moves quickly between the attacks.

Since the attacks were against the routers at the data center rather than against a specific domain or server, it was impossible for us to know which sites were being affected. For example, your sites were all down to us since our requests were being routed to one of the routers under attack.

We were trying to give the sites that had previously affected by the outage a few was ago the highest priority in the move queue which is why your sites were moved last night.

We know how frustrating this outages are and we are doing everything we can to keep them from reoccurring. We are sorry that we weren’t able to respond as quickly as you are used to.

Please let us know if there is anything else we can do. Thank you again!

I got additional details from another staffer, Ken Spreitzer:

I wanted to jump in and add something else to this. Everything Nami explained about how we had to move sites quickly is absolutely the case. But I wanted to add some extra detail.

We move sites between servers using a very advanced script that makes sure everything is copied correctly and that there is (usually) no downtime. In a normal circumstance, you would not have even noticed any downtime at all (which is why we normally don’t need to give any advanced warning of a move). However, in the middle of moving your sites last night, Hurricane Electric experienced another brief network attack. This prevented our script from completing normally. We detected this, of course, and manually completed the copy. It was during this manual copying that we accidentally setup your aliases incorrectly.

This all only happened because of an incredibly rare coincidence of timing that we couldn’t have predicted (an unexpected outage during a split-second moment of critical timing in our script), followed by a manual mistake. For what it’s worth, yours is the only account that this happened on. We know that may not be much consolation, but at least wanted you to have a better idea of our process and why it happened. Again, we apologize for the inconvenience.

As always, please let us know if you have any questions or if there is anything else we can do.

Their prompt and human follow-up is another reason why I’ve hosted all my sites (and most of my clients’ sites) at TigerTech for years.