CloudFail.net

We are currently investigating internal network connectivity between the Miami-b and Miami-c data centers.

All webbies are up and accessible from their external IPs but management services are currently affected.

Customers in the Miami-c data center will be unable to use Webby Manager to reboot or redeploy their webbies until the network connectivity is restored.

All webbies in the Miami-b data center are NOT affected by the connectivity loss.

UPDATE: All connectivity has been restored. Please open a ticket if you are having any problems.

We’ve identified a network issue on Dallas-02. We are investigating the cause and working to fix asap. Please stand by.

We had an issue with our internal xen network bridges. Issue has been fixed and we’re back.

Note: All Webbies remained running.

We are experiencing an outage on Miami-b11. We are having critical issues with a storage array. Please hold while we work on bringing this server back.

Update: Issue is still being worked on. We will post more details soon.

We are still working to find the cause of the brief network outage in Miami. The onsite admins have restored connectivity. There was no downtime for webbies, only a loss of network connectivity.

UPDATE We thought things had been resolved, but the onsite admins are still working on the outage, bear with us, we’ll have things back online shortly.

UPDATE Our onsite admins have reported that we are 100% back. Outage has been resolved. Please open a ticket if you need any further assistance.

We are currently performing emergency maintenance on Miami-b07. We are checking the file system integrity to ensure no data loss.

More updates to follow.

UPDATE All maintenance is finished. If anyone has any problems, please submit a ticket.

We are currently performing more patch and security updates on nodes in the Wash DC data center. Wash-07 is being rebooted to apply patches. Updates to follow.

UPDATE 17:15 EST Wash-07 is coming back online and all webbies are now booting back up. If anyone sees any issues once all webbies are online, please submit a support ticket and we’ll handle any issues.

We are currently performing more patch and security updates on nodes in the Wash DC data center. Wash-06 is being rebooted to apply patches. Updates to follow.

UPDATE 13:20 EST Wash-06 is back online and all webbies are coming back up.

We are rebooting Wash-01 && Wash-04 to remediate some needed updates and system patches. There will only be a brief downtime. We’ll update here once finished.

UPDATE 16:45 EST Webbies are now coming back online on Wash-01

UPDATE 17:20 EST Webbies on Wash-01 should all be back online. Wash-04 is coming back up now.

UPDATE 17:27 EST All webbies on Wash-04 should now be back online.

If anyone is seeing any problems, please submit a ticket.

We are currently checking on a connectivity issue to Wash-04. We’ll update once we know more.

UPDATE 14:20 EST We’ve identified a NIC port caused the system to lock up and we initiated a reboot to bring things back online. We are investigating if the problem is related to the switch port the node is connected to or the NIC card.

UPDATE 14:35 EST All webbies are coming back online, please open a ticket if you are still experiencing any problems.

We have mitigated a DoS Attack that was affecting network connectivity in the Miami datacenter. DoS Attack has been mitigated and all connectivity is back to 100%.

If you experience any further issues please get in touch with support.