Service updates for popular cloud hosts.
07:00 AM GMT – 15:00 AM GMT – We are going to update OnApp software on these clouds. Power action buttons might be temporarily disabled within this time.
This is done. Thank you for your patience.
5:31PM MST – We have noticed a post onapp upgrade network related issue. Our admins are currently working to fix this.
At approximately 1:15 PM CT, the Rackspace Cloud Operations team identified a network traffic issue within DFW1-2 of our DFW Data Center that is affecting multiple clusters (SSL, PHP, etc.). Currently, customer sites may be down or experiencing latency in their connectivity. Our Operations team is actively working to resolve the issue.
UPDATE: As of 2:45 PM CT, our engineers and technicians are working to correct this issue.
UPDATE: As of 2:55 PM CT, our engineers seem to have corrected the issue, we will continue monitoring the system.
UPDATE: As of 3:55 PM CT, our engineers are still seeing some sites as intermittently being affected. We will continue to monitor and work on this issue.
UPDATE: As of 5:45 PM CT, our engineers and technicians are still working to correct this issue.
UPDATE: As of 6:35 PM CT, our engineers and technicians are still working to correct this issue.
UPDATE: As of 7:45 PM CT, our engineers and technicians are still working to correct this issue.
UPDATE: AS of 8:00 PM CT, our engineers have resolved the issue. We apologize for any issues during this time.
Thank you for your patience during this matter. If you have any further questions, please contact a member of our support team by visiting us in live-chat or by calling 1.877.934.0407 or INTL+1.210.581.0407.
We’ve got campfire back online. Sorry about that!
We’ve identified the source of the problem. Should have things back online in just a minute.
Campfire is intermittently broken. We’re working quickly to fix the problem.
SoftLayerNotify: #909051 – 10:26 05-Feb-2012 UTC: We have recovered services for customers behind fw01.fcr02.wdc01 at this time.
SoftLayerNotify: #909049 – 10:03 05-Feb-2012 UTC: One of our multi-tenant firewall devices in WDC01 is experiencing troubles, we are working to recover it.
Dear customers. We are experiencing some slowness on some Atlanta VPSes, including all Cloud Hosting servers located on Atlanta cloud. Our L3 team is working on this problem. We apologise for the inconvenience.
[Update]
Feb 04 20:22 – We had to start the file system integrity check on one of our Cloud Hosting servers. This process is going to take about 2 hours.
Feb 04 22:07 – Due to some problems appeared with the new disks, we are going to move all our Cloud Hosting servers to the old disks. The servers will be rebooted. We apologize for the downtime caused by the reboots.
Feb 04 22:38 – Email sent to customers:
Dear Customers,
As you may be aware there was a failure on the cloud hosting platform earlier today. This appears to be linked to a filesystem issue on the cloud storage. As such we have taken the decision to roll back to a backup of the cloud hosting platform that is 2 days old to allow sites to be restored.
** We aim to restore the full cloud platform, with current data, within 12-24 hours **
As a temporary measure we have blocked incoming email to avoid any new emails getting lost. Any pending emails will be stored temporarily on backup email servers. Once the cloud platform is fully restored we will allow these emails to flow into the restored service. Whilst we do understand the importance of email we have taken this decision to make sure no email is lost or left out of sync. All other functions (website/control panels) are up and running.
We apologise for this outage, and are working as quickly as possible to restore service.
Once this is restored we will be sending a full RFO (Reason for Outage) to customers, including any credits that are due under the SLA.
Should you have any questions about this please don’t hesitate to contact us on supp...@vps.net.
–
[Update] Feb 05 15:20 GMT
We are still working to restore one of Cloud Hosting servers. All others have been restored.
[Update] Feb 05 21:40 GMT
We are expecting to finish restore in next 10 hours.
[Update] Feb 06 07:30 GMT
The issue with Cloud Hosting server should be fixed and all sites should be back.
If you see any issues with your sites on Cloud Hosting server, please create ticket and support team will check it.
We’ve got Backpack serving up files happily again. We are sorry for the prolonged nature of this issue. It took us an unusually long time to debug this one despite our best efforts.
Dear customers. Our backup SAN on Amsterdam cloud is unreachable. We strongly recommend you do not try to restore from backups or create new backups until issue will be resolved. We apologise for the inconvenience.
[Update] Monday 06.02.12 18:30 GMT
The Backup SAN is already reachable.
All issue have been fixed.