What the hell happened?
Whew, what a day.
A little over 24 hours ago there was a massive power outage in Orlando, partially due to the passing weather from Hurricane Zeda. They use a very sophisticated backup power generator when situations like this happen that kicks in with a bridge that transfers power from the grid to the backup generators without allowing for any loss of server uptime. It's the failsafe transfer of power.
That bridge between power sources blew up while making the transition taking the whole backup system generator with it. No data was lost anywhere, but all of the redundancies that were built into this system that have been running for 23 years without incident collapsed into darkness while making the transition.
The sites are coming back online, but not all at once. DNS settings will need to re-propagate and that's not a linear process so you might see some inconsistencies as mail servers and website servers are brought back online. What we are already seeing this morning is most sites are live again but not back to 100% performance. E-mail servers are still coming back online. We're showing most of our sites and systems back online as of 8am CST. ApexSpeed still does not have e-mail working.
TL;DR:
A perfect storm of a massive power outage and a backup power system failure took down our host provider's system and backup power yesterday morning. This also blew up redundant systems simultaneously destroying a lot of hardware in the process. No data was lost though and websites and e-mail are coming back online as we speak. We're almost completely back to 100% and we apologize for the loss of a day of uptime.
It's not fully back to speed, but we're getting there. Thanks for sticking with us through this. Hope the withdrawal wasn't too bad! It was a helpless feeling all day yesterday while we couldn't do anything but wait. Glad to be back.
Doug & Kim
Reply With Quote
Originally Posted by whadeduck
