CloudFail.net

April 27 2012 07:00 GMT ATL-D, ATL-E: We are starting maintenance on the network switches for both the ATL-D and ATL-E clouds. All VPSs will be powered down during this work. CloudHosting service in Atlanta is also affected by this work. These services should be available again in 3 hours.

UPDATE: April 27 2012 08:50 GMT work on Atlanta-E is progressing, completion ETA: 1.5 hours. Following this maintenance, ATL-D cloud and Atlanta CloudHosting will be taken offline for similar work.

UPDATE: April 27 2012 09:40 GMT Atlanta-E is still offline for this network maintenance, we should be initiating the startup of VPSs on this cloud (ATL-E) in 1 hour. Due to the duration of the Atlanta-E maintenance, our engineers will be postponing Atlanta-D network maintenance.

UPDATE: April 27 2012 11:15 GMT There is an delay with the final completion of the work on this cloud. Our Engineers are working with on-site DataCenter staff to finish our work. All work on Atlanta-D is postponed.

UPDATE: April 27 2012 12:10 GMT One final step is preventing the complete restoration of service for this cloud. VPS.net L3 Engineers are continuing co-ordination with DataCenter Technicians to complete this work.

UPDATE: April 27 2012 14:53 GMT All VM’s are booting now.

UPDATE: April 27 2012 15:23 GMT All VM’s have been booted. If yours is not online, please open a ticket or email supp...@vps.net. Thanks for your patience.

April 26 2012 6:25 GMT We are experiencing issues with SAN3 on LON-B. Level 3 support is currently looking into this issue right away.

UPDATE: April 26 2012 10:28 GMT Once disk has finished rebuilding and we are currently working on disk replacement for the other.

UPDATE: April 26 2012 11:03 GMT Second disk has been replaced and we are now rebuilding it.

UPDATE: April 26 2012 12:04 GMT The SAN is back to a normal state and is rebuilding. The progress of the rebuild is at 22% and VMs are starting.

UPDATE: April 26 2012 13:40 GMT The progress of the RAID rebuild is at 44%, all VMs are on-line and available.

UPDATE: April 27 2012 22:38 GMT The RAID rebuild of SAN3 is now at 92% complete

UPDATE: April 27 2012 22:38 GMT The RAID rebuild of SAN2 is now at 76% complete

UPDATE: April 28 2012 7:40 GMT The RAID rebuild of SAN3 is now at 95% complete

UPDATE: April 28 2012 7:40 GMT The RAID rebuild of SAN2 is now at 97% complete

UPDATE: April 29 2012 08:55 GMT SAN2 and SAN3 are offline due to an error with the rebuild. All affected VPSs on these SAN are offline. Our L3 engineers are working to get these SAN’s online again. They will then be able to startup all affected VPSs again.

UPDATE: April 29 2012 11:00 GMT Issues on SAN4 are resolved and those VPSs should be started again soon.

UPDATE: April 29 2012 12:30 GMT SAN4 is online but rebuilding following a disk replacement, performance will be slower. SAN3: restarted rebuild, it is at 21%, but all VPSs on SAN3 are currently offline. SAN2: We will be able to fix this after the SAN3 rebuild is complete.

UPDATE: April 29 2012 13:09 GMT Rebuild on SAN4 is complete and all VMs have been started. SAN2 rebuild at 29%, SAN3 rebuild at 87%. All VMs on SAN2 and 3 are powered off.

UPDATE: April 29 2012 19:43 GMT SAN2 rebuild at 98% and SAN3 rebuild at 98%.

UPDATE: April 29 2012 21:35 GMT SAN2 has been repaired. Virtual machines are now booting. We expect this process to take about 90 minutes.

UPDATE: April 30 2012 0:18 GMT All VMs from SAN2 should now be started, if you are still experiencing isssues please contact supp...@vps.net. VPSs from SAN3 are being migrated to a new SAN.

UPDATE: April 30 2012 04:25 GMT SAN2 Rebuild is 72% complete. VPSs from SAN3 are continuing to be migrated to other SANs. Power actions on this cloud continue to be disabled, please create a support ticket if you need any Power actions performed by our team.

UPDATE: April 30 2012 09:20 GMT SAN2 Rebuild is 74% complete. Migration of VPSs from SAN3 is almost complete.

UPDATE: April 30 2012 13:05 GMT Power actions are now enabled

April 17 2012 13:31 GMT We are experiencing issues with some HV’s on this cloud. Our level 3 technicians are working on resolving this issue.
Update: April 17 2012 17:35 GMT The HV issues have been fixed. Should you have any further issues please contact support.

April 12 2012 2:30 P.M. MDT – We are experiencing slow performance on LUN3 of our Atlanta-E cloud. Our L3 engineers are working on this issue.

April 13, 2012 13:00 GMT – L3 techs are still working with that LUN to fix the issue.
April 13, 2012 17:52 GMT – L3 techs are still working on the problem. ETA ~ 60 minutes.
April 13, 2012 18:08 GMT – L3 techs are starting all VM’s now. ETA ~ 45 minutes for all VM’s to be started.
April 13, 2012 19:27 GMT – VM’s have all been started.
April 13, 2012 19:51 GMT – We are processing some VM’s that have failed to boot, ETA: 20 minutes.
April 13, 2012 21:42 GMT – It has become necessary to power down the cloud for emergency network maintenance. ETA: 1 hour
April 13, 2012 21:57 GMT – VM’s are starting up now.
April 13, 2012 22:10 GMT – VM’s are all back online.

April 10, 2012 Our Atlanta 2 cloud hosting server is not accessible right now. Our techs are working on resolving this issue. A file system check is currently running.

Update: The file system check has finished and the server is back online. Should you have any further issues please open a ticket.

We have lost connection to London cloud G due to power loss. DC techs are working on restoring it

UPDATE:
5:11 AM GMT Power has been restored. Waiting for hypervisors to come up online
8:49 AM GMT Almost all services in LON-G have been restored. VMs are online. Should you have any questions feel free to contact our support team.
9:49 AM GMT All issues have been solved.

Our goal is to provide our clients with a robust and fully redundant network enabling high availability and uptime for the services we offer. To achieve that goal it is necessary to schedule a network maintenance window which will introduce latency and brief periods of interruption. Our efforts to make proactive and controlled changes will significantly reduce the risk of unplanned events.

Maintenance Details
Location: Salt Lake City, UT data center
Start Date & Time (MDT): Saturday, March 31, 2012 10:00 p.m.
End Date & Time (MDT): Sunday, April 1, 2012 2:00 a.m.
Description of Work: Network engineers will adjust routes to the Internet to improve network performance. During the maintenance window there will be several periods of intermittent latency and brief periods of interruption.
Services Affected: All services within the data center facility.

Our Backup SAN on the SLC-C cloud is currently offline.
VPS.net L3 Engineers are currently working to repair this SAN.
Do not restore your VPSs on this cloud from snapshot backups

Update
March 27, 6:35 AM GMT We have performed file system check and backup SAN is now back online

We are experiencing a performance issue on one of our SANs on this cloud.
Our L3 Engineers have been working to isolate the cause of this slowdown and repair it.

March 23, 19:00 GMT: Our Cloud Hosting server has some issues and it not available right now. Our techs are working to bring it up as soon as they can.

March 24, 04:00 GMT: We are rebuilding Cloud Hosting server. When it is up, we will start to restore all sites there.

March 24, 05:00 GMT: We are restoring all customers’ data (sites, DBs, mailboxes, etc)

March 24, 11:30 GMT: We have finished the restoration of all data