We’ve experienced a localized power failure in our Chicago data center that is impacting DNS service for a subset of our customers. We have technicians on site and have restored a portion of the impacted services, and are actively working to restore all remaining impacted services as soon as possible.
We have isolated the issue to one of our three uninterruptible power supplies (UPS), which suffered a critical fault and triggered safety breakers. Bypassing the affected systems could have resulted in electrical damage to the impacted servers. Our teams have been working on two options: Firstly, getting alternate power to the affected racks; secondly, we have electricians exploring a safe way to work around the damaged UPS.
A portion of the affected customers are back online. The team continues to work on getting the rest of those customers that were impacted back online, as well as some of our customer-facing services.
All impacted customer DNS services are back online. Team is now focused on remaining impacted websites and mail systems, including POP, IMAP and Exchange. We are working to provide an ETA on their availability.
Power has been restored to affected racks. Team is working to ensure that systems come back online as they go through the boot / startup process. DNS systems are currently under heavy load due to increased and queued traffic. Performance should stabilize as temporary spiked volume subsides.
Two steps forward, one step back: Just had a power cycle on two racks during UPS repair, causing some systems to start the reboot process again. Power is active to those racks; it’s just a matter of the systems booting again.
Regarding email: There should be no loss of actual email messages or data for any impacted customers. Inbound messages should be queued up, and you should receive a batch of messages including those sent during the service interruption as soon as service is restored.
Impacted mail servers are back online and quickly churning through the backlog. Affected customers should start to see inbound email activity over the next 60 minutes.
We’re working now on getting SiteControl back up and fully functional. We’re now also investigating individual servers based on customer feedback here and through the phone lines to zero in on any lingering functionality issues.
The email backlog is pouring through our mail servers. We estimate that all backlogged emails will have have passed to their destinations in between 1 and 2 hours.
For those still having password authentication issues: Your password has not been changed or lost. The database with that information is syncing to accounts now, and that issue should be resolved shortly.
Customers with Shared SQL servers should all be back online now. SiteControl should also be online for all affected customers.
If you are still having issues with your website, or with logging into your email, please contact 866-467-8929 to get individualized assistance. The queue has diminished significantly.
Our technicians continue to work on our Exchange 2010 environment, which is still not working for some users. We will provide updates when able.
The majority of email transport and routing issues are now resolved, and the majority of mailbox databases have been mounted; we are in the process of dealing with the problem ones now. Translation: The majority of Exchange 2010 users should now be back online. However, there are some exceptions; we’re working as quickly as we can to resolve any issues for the remaining users.
All Exchange mailbox servers are back up and running now. Please contact us if your service has not resumed and your emails from the outage have not yet started to come through.
During the recovery process for Exchange 2010 services, we experienced some database corruption for a subset of customers using Exchange 2010. We are working to rebuild and restore affected mailboxes. The remaining databases will take time to rebuild due to their size, so an exact ETA is unpredictable at this moment, but we will continue to provide updates until full resolution. In the meantime, users should be able to connect to the server to send and receive email, but won’t have access to older emails until the DB is fully repaired. Once complete, all older email should be present. We are working with the utmost urgency to restore all services.
More than 95% of all the mail databases on the Exchange cluster have been rebuilt. The remaining ones are the largest in size, and could take some time. All customers should be able to send and receive email. Once the final rebuilds are complete, all stored email should be accessible. Our support team is proactively reaching out to the remaining affected customers.
21 of 33 databases are now ready to be mounted. We are still cleaning up the other 12. We’ll soon start to link users’ mailboxes to their originals, then merge new email to original databases tonight. At that point, customers will see old and new email data together. This merger will cause a service disruption of 5-10 minutes.
Quick update on the remaining mail databases: 12 databases were mounted overnight and are fully functional. Some of the customers whose databases were remounted are reporting connection issues; the engineering team is now investigating this. Another 11 are ready to mount. The largest 10 databases are still undergoing repairs. Mailboxes residing on fully fixed databases should be able to see both old and new email data. We thank you for your patience in our continuing efforts to restore your service back to normal.
Update on the remaining Exchange databases: 21 of the 33 damaged databases are fully functional. Another eight are ready to mount. Four databases are still undergoing repairs.
32 of the 33 affected Exchange databases have been repaired. If you are experiencing any issues related to your mailbox, please contact support, as hold times are significantly reduced.
All Exchange databases have been repaired and mounted.