CloudFail.net

We have finished migrating all nodes in the Miami-B site on to the new power grid. Each node was turned off, circuits moved over, and started back up. All Webbies should be up, please get in touch with Support via the Webby Manager if your Webby is still down.

Sorry for this hassle, but we have to take this precaution in order to avoid further power issues.

Thanks,

Carlos.

We are currently performing an emergency maintenance on miami-b08. At this time we are working on moving everyone off of miami-b08 to a new node. All Webbies are safe, we are taking all the necessary steps to ensure no data loss.

Manager is down for maintenance. We will post back with progress or a completion update.

Spoke too soon. Maintenance is completed early.

Please let us know if you are having any issues.

We’re currently experiencing network failure from within our datacenter, we have technichans over to check it out, please stand by. We’ll be updating this blog post.

Our datacenter had a power surge. All Webbies are getting back online. Please open a ticket in case you’re still having problems.

UPDATE 5:07PM EST: Only webbies in nodes miami-b06 and miami-b11 are still down. Webby Manager is still down as well. We have all hands on deck working to fix this.

UPDATE 5:17PM EST: miami-b11 is back up. Working on getting miami-b06 back up.

UPDATE 6:10PM EST: Webby Manager is running again. We are having problems with the host system in miami-b06 so please bare with us for a little longer.

UPDATE 8:20PM EST: We have just finished verifying all data on miami-b06 and all Webbies are now starting to come back online. Please open a ticket if you are still experiencing any issues.

[0:30AM] We’re having connectivity issues with Webbies in the 67.23.79.* range. We have all hands on deck working to fix the problem.

[0:45AM] It was a Level3 Connectivity problem that is now resolved.

All Webbies are up and running.

[07:30PM EST] There was some packet loss at Atlanta’s Level 3 router. A ticket was opened and the problem is now solved. All Webbies remained up during the network outage.

[1PM EST] Our DC had an UPS overload and that caused a reboot of nodes miami-b01, b02, b03, b12, b13, b15 and b16. All webbies on those nodes were rebooted. Approximate downtime was 7 minutes and all Webbies are back up.

Please open a ticket if you are still experiencing problems.

[11:50 PM EST] We are working with our network provider level 3 to solve a routing issue that has caused some packet loss for traffic going through this route.

Update: 12:03AM Issue has been resolved.

[11:40 PM EST] We are working with our network provider level 3 to solve a routing issue that has caused some packet loss for traffic going through this route.

Update: 11:59 PM EST Issue has been solved, we’re back to optimal level3 performance.

We had a raid failure on miami-b12. The server is up and running, however the raid is running a full rebuild, so the disk access might be slower than usual.

We will keep monitoring the progress, and update this post.

Note: All Webbies are up.

Update 11PM EST: The rebuild has fully finished with no problems.