Cloud Fail

[11:40 PM EST] We are working with our network provider level 3 to solve a routing issue that has caused some packet loss for traffic going through this route.

Update: 11:59 PM EST Issue has been solved, we’re back to optimal level3 performance.

We had a raid failure on miami-b12. The server is up and running, however the raid is running a full rebuild, so the disk access might be slower than usual.

We will keep monitoring the progress, and update this post.

Note: All Webbies are up.

Update 11PM EST: The rebuild has fully finished with no problems.

We are suffering from an issue where GitHub’s API is timing out. This is affecting any deployments that uses GitHub integration.

We’re actively working on this problem however we don’t have an ETA at this time.

Thanks for your patience!

Update 9:30AM EST: It seems it was a hiccup on GitHub’s end. Integration is available again.

[9:17PM EST] We hit a Xen bug with miami-b04, we will update this post as soon as its finished.

Update 9:22AM We had to reboot miami-b04 and Webbies are coming back online.

Please contact support if you have any issues.

We had a raid failure on miami-b11. We had to turn off the server for a hardware maintenance, and it is now complete.

The server is back up and Webbies are coming back online.

If you have any issues please contact support.

UPDATE 8:20PM :

We continue to have issues with miami-b11 we are now working on migrating everyone off miami-b11. We’ll update this shortly.

UPDATE 01/24 0:20AM :

All Webbies are up but we have disabled rebooting, stopping, starting and redeploying Webbies in the meantime. Once you’re migrated to another node, you’ll be ready to manage your Webby as usual.

Update 01/24 8:00PM :

We have finished migrating all Webbies to a new node. We managed to rescue all data, no data loss was suffered.

[3:15PM EST] We are working on an issue with miami-b04, we will update this post as soon as its finished.

[4:15PM EST] Miami-b04 is now back. We had to do an emergency hardware maintenance and no data was lost.

We apologize for the long delay, however we had to do this carefully to avoid any possible data loss and more downtime.

All Webbies are back now, if your Webby isn’t up, please contact support.

1:35AM EST. After applying some kernel patches to a group of nodes, we had to perform reboots on them. All nodes are back up and Webbies are coming back online.

Please contact support if you experience any issues.

1:19 PM EST: We are investigating Xen problems on miami-b10.