CloudFail.net

Date: 5/16/2012
Original Time: 9:00PM EST

New Time:  10:00PM EST.

At 9:00PM EST 10:00PM EST on Wednesday May 16th 2012, we will be doing a planned maintenance on Webbynode server miami-c03. We need to shutdown the server to perform a hardware replacement within the server. This maintenance window should be short, we estimate it will take around 15 minutes, but we need to schedule it anywhere from 9 to 10PM EST.

Best

Carlos Taborda

We are experiencing issues with miami-b05. Please stand by as we identify the source of the problem.

- Update 11:45AM – We experienced a hardware failure on miami-b05. The server has been unracked and we are working on it.

-Update 12:20PM – The server is back up online. We had some dead RAM chips and we replaced them and we’re back online. All Webbies should be coming up now, please let us know if you are still experiencing issues with them.

Thanks for the patience.

Hello,

As of this morning we are having an issue with miami-b01. Some customers have been experiencing some performance issues, and we are checking in to it. We may be doing some restarts and the server will be unavailable as we work on it.

Update: 10:53AM – We are facing a corrupt OS partition. We are taking all preventive measure to make sure we have no data loss.

Update: 11:59AM – We’re still working on bringing the server back. The server is still unavailable as it is unracked while we work on it.

Update: 12:23PM – We have been able to get to all logical volumes and group. We are now running FSCK commands to bring back the OS partition.

Update: 1:20PM – We are having to re-install the OS without disturbing the data for the Webbies. After this completes we will have to install our internal management software.

Update: 4:00PM – Webbies are coming back online. We are finalizing some issues with a few webbies but the majority of webbies are now back online. We are working on bringing the rest online.

Update: 5:30PM – All Webbies are back online. We were able to get this server restored without any data loss with success. If you are still down please let us know so we can check it out.

We are investigating a network issue with everyone who is on the 67.23.79.x ip range.

Update: 2:34PM IP Range is back online.

We are currently experiencing some issues with miami-c05 and it will require a reboot. We will update this ticket as soon as it is back up.

Update: 615PM EST. The issue is with our storage array. We are troubleshooting it at this moment.

Update: 630PM EST. We are still working on bringing it back. The Webbies are booting while the server is on our work bench. We will be racking it back as soon as everything seems stable.

Update: 714PM EST. The server is back up and Webbies are coming back online at this moment. Please standby.

We are currently investigating internal network connectivity between the Miami-b and Miami-c data centers.

All webbies are up and accessible from their external IPs but management services are currently affected.

Customers in the Miami-c data center will be unable to use Webby Manager to reboot or redeploy their webbies until the network connectivity is restored.

All webbies in the Miami-b data center are NOT affected by the connectivity loss.

UPDATE: All connectivity has been restored. Please open a ticket if you are having any problems.

We’ve identified a network issue on Dallas-02. We are investigating the cause and working to fix asap. Please stand by.

We had an issue with our internal xen network bridges. Issue has been fixed and we’re back.

Note: All Webbies remained running.

We are experiencing an outage on Miami-b11. We are having critical issues with a storage array. Please hold while we work on bringing this server back.

Update: Issue is still being worked on. We will post more details soon.

We are still working to find the cause of the brief network outage in Miami. The onsite admins have restored connectivity. There was no downtime for webbies, only a loss of network connectivity.

UPDATE We thought things had been resolved, but the onsite admins are still working on the outage, bear with us, we’ll have things back online shortly.

UPDATE Our onsite admins have reported that we are 100% back. Outage has been resolved. Please open a ticket if you need any further assistance.

We are currently performing emergency maintenance on Miami-b07. We are checking the file system integrity to ensure no data loss.

More updates to follow.

UPDATE All maintenance is finished. If anyone has any problems, please submit a ticket.