CloudFail.net

February 8, 2012 12:40 PM GMT There is an ongoing SAN synchronization at London cloud I. Some VM’s may experience degraded disk performance.

10:00 AM GMT – 15:00 AM GMT – We are going to update OnApp software on these clouds. Power action buttons might be temporarily disabled within this time.

The part of virtual machines is not accessible because of update in London E.
We are working on this issue right now

10:00 AM GMT – 15:00 AM GMT – We are going to update OnApp software on these clouds. Power action buttons might be temporarily disabled within this time.

February 6, 05:38 PM MST
We are experiencing performance issue on one of the SANs as it has RAID rebuild in progress. This may cause low disk performance on some VMs. ETA 1hour

[Update]
Feb 07 09:23 – We are starting VPSs now.
Feb 07 10:33 – The RAID is still being rebuilt. It’s 8% now.
Feb 07 14:34 – The RAID is still being rebuilt. It’s 20% now.
Feb 07 19:19 – The RAID is still being rebuilt. It’s 29% now.
Feb 07 19:46 – The RAID is still being rebuilt. It’s 33% now.
Feb 07 22:30 – The RAID is still being rebuilt. It’s 40% now.
Feb 07 23:30 – The RAID is still being rebuilt. It’s 43% now.
Feb 08 03:30 – The RAID is still being rebuilt. It’s 54% now.
Feb 08 04:30 – The RAID is still being rebuilt. It’s 57% now.
Feb 08 07:25 – The RAID is still being rebuilt. It’s 64% now.

07:00 AM GMT – 15:00 AM GMT – We are going to update OnApp software on these clouds. Power action buttons might be temporarily disabled within this time.

This is done. Thank you for your patience.

5:31PM MST – We have noticed a post onapp upgrade network related issue. Our admins are currently working to fix this.

Dear customers. We are experiencing some slowness on some Atlanta VPSes, including all Cloud Hosting servers located on Atlanta cloud. Our L3 team is working on this problem. We apologise for the inconvenience.

[Update]
Feb 04 20:22 – We had to start the file system integrity check on one of our Cloud Hosting servers. This process is going to take about 2 hours.
Feb 04 22:07 – Due to some problems appeared with the new disks, we are going to move all our Cloud Hosting servers to the old disks. The servers will be rebooted. We apologize for the downtime caused by the reboots.
Feb 04 22:38 – Email sent to customers:

Dear Customers,
As you may be aware there was a failure on the cloud hosting platform earlier today. This appears to be linked to a filesystem issue on the cloud storage. As such we have taken the decision to roll back to a backup of the cloud hosting platform that is 2 days old to allow sites to be restored.

** We aim to restore the full cloud platform, with current data, within 12-24 hours **

As a temporary measure we have blocked incoming email to avoid any new emails getting lost. Any pending emails will be stored temporarily on backup email servers. Once the cloud platform is fully restored we will allow these emails to flow into the restored service. Whilst we do understand the importance of email we have taken this decision to make sure no email is lost or left out of sync. All other functions (website/control panels) are up and running.

We apologise for this outage, and are working as quickly as possible to restore service.

Once this is restored we will be sending a full RFO (Reason for Outage) to customers, including any credits that are due under the SLA.

Should you have any questions about this please don’t hesitate to contact us on supp...@vps.net.

–

[Update] Feb 05 15:20 GMT
We are still working to restore one of Cloud Hosting servers. All others have been restored.

[Update] Feb 05 21:40 GMT
We are expecting to finish restore in next 10 hours.

[Update] Feb 06 07:30 GMT
The issue with Cloud Hosting server should be fixed and all sites should be back.
If you see any issues with your sites on Cloud Hosting server, please create ticket and support team will check it.

Dear customers. Our backup SAN on Amsterdam cloud is unreachable. We strongly recommend you do not try to restore from backups or create new backups until issue will be resolved. We apologise for the inconvenience.

[Update] Monday 06.02.12 18:30 GMT
The Backup SAN is already reachable.
All issue have been fixed.

07:00 GMT – 15:00 GMT – We are going to update OnApp software on these clouds. Power action buttons might be temporarily disabled within this time.
This is done. Thank you for your patience.

[3pm MST] We are having maintenance on our Atlanta Cloud Hosting servers. All the data is currently moving to more stable SANs.

[4:06pm MST] Everything should be up and back to normal. We are sorry for any inconveniences caused.

Thank you for your patience, and business!