Cloud Fail

On
the
morning of Friday, March 12, The Rackspace Cloud system
administrators will be performing routine maintenance across our MySQL
database infrastructure in our Dallas (DFW) data center.  The
necessary maintenance tasks will be performed on each MySQL cluster in
DFW  between 01:00 am and 01:45 am CST.  During the maintenance window,
some clients may see brief connectivity issues.  We expect this
maintenance to have very minimal impact.

As of 8:48 PM CST, The Rackspace Cloud team has determined that there is an issue with the Cloud Server access in the control panel.  At this time we have our development team researching the issue with no ETA  on resolution.  Please check back to this post for updates.  We apologize for this inconvenience and
will update this blog post as we receive more information. 

Update: Currently as of 5 AM CST, The Rackspace Cloud team are still working on the issue with the Cloud Control Panel.

Update: As of 6 AM CST, The Rackspace Cloud team has corrected the issues with the Control Panel.

If you have any questions or concerns please Contact our support team via live chat or at 1-877-934-0407.

As of 11:55 AM CST, The Rackspace Cloud engineers have identified an issue causing degraded performance on MySQL cluster 50-06 in our WC1.DFW1 data center.  You may see slow performance or sites not load at all at this time.  We will continue to update this blog as more information becomes available. 

UPDATE: As of 12:26 PM CST, The Rackspace Cloud Engineers have corrected the issue causing degraded performance on the MySQL 5–06 cluster.  If you have any questions or concerns please contact our support via
live chat or at 1-877-934-0407 international +1.210.581.0407.

As of 11:20 AM CST, The Rackspace Cloud’s domain registration service OpenSRS is temporarily in a degraded state. Customers will be unable to register domains until services have been restored.  We will continue to make updates to this post as they become available.

UPDATE: OpenSRS has corrected the issue with their domain registration service.  You many now once again register domains through your Rackspace Cloud account.

As of 9:35 AM CST, The Rackspace Cloud engineers have detected an issue causing degraded performance in our ASP1-5 cluster in our WC2.DFW1 Cloud Sites data center.  This is an isolated issue affecting only sites on this single node.  Sites on this node will load slowly or not at all.  We will continue to report status updates as they become available.  If you have any issues, please do not hesitate to contact a
member of our support team via live-chat or at the following
telephone numbers: 24-hour toll free 1.877.934.0407 and INTL
+1.210.581.0407.

UPDATE: As of 9:43 AM CST, service has been restored to our ASP1-5 cluster.  Thank you for your patience as we worked to correct the issue.  If you have any other concerns, please visit with a member of our support staff.

On March 2nd from 03:00am – 06:00am CST, The Rackspace Cloud system administrators will be
performing priority maintenance on two of our MySQL database clusters in
our SAT data center.

During the
maintenance window some SQL queries utilizing these clusters will be in a
read locked state (INSERT, UPDATE, DELETE, ALTER queries will be
ignored) for the duration of the maintenance.  There may be a period of
up to 10 minutes in which databases will be unavailable.

We
expect this maintenance to last no longer than the time listed.  We
apologize for the inconvenience, but we need to complete this work in a
timely manner.  Please note that no other technologies will be effected
by this maintenance.

UPDATE: Cloud Engineers have rolled back maintenance changes and the servers are back online.

The Rackspace Cloud’s system engineering team will be performing control
panel maintenance on Tuesday, March 2nd, 2010 from 10:00 PM to
11:00 PM CST.  During this maintenance the control panel will be temporarily offline.  We expect this maintenance
to be completed within this time window and as always work to keep
maintenance down to a minimum. 

UPDATE:  As of 10:41 pm CST, Our Rackspace Cloud Engineers have completed the maintenance and everything should be back online.

If you have any issues, please do not hesitate to contact a
member of our support team via live-chat or at the following
telephone numbers: 24-hour toll free 1.877.934.0407 and INTL
+1.210.581.0407.

As of 11:15 AM CST, the Rackspace Cloud system engineers have detected
an issue that’s causing degraded performance in our DFW Data center.  Our team is actively working to resolve the issue, any updates we have on this issue will be posted here.

If you have any further questions, please contact a member of our
support team by visiting us in live-chat or by calling 1.877.934.0407
or INTL +1.210.581.0407.

Update: As of 11:53 am CST, the Rackspace Cloud system engineers have resolved the issue and normal functionality has been returned to the DFW Datacenter.

The Rackspace Cloud’s system engineering team will be performing several upgrades to our WC2 infrastructure in our Dallas / Fort Worth, TX (DFW) data center to enhance stability and performance.  The maintenance is scheduled to last two (2) hours, beginning on the morning of March 6 at 12:00 AM CST and ending at 02:00 AM CST.  While connectivity during this window will be intermittent, some sites may be offline for as long as 90 minutes.  As always, we will do everything we can to keep the interruptions to an absolute minimum.



STORAGE ARRAY MAINTENANCE

During this maintenance window we will be performing a series of software upgrades and disk space additions to our storage array to increase capacity, stability, and scalability.  These upgrades will also allow us to further optimize our storage array’s performance, equating to faster data load times for your applications and better stability.  During the maintenance period you may be unable to access, upload, or make changes to your website content.   We will certainly do everything possible to limit the amount of time your account will be affected.



We apologize for the short notice of this maintenance.  It is critical, however, that the work be completed in as timely of a manner as possible to ensure increased stability and scalability.  This maintenance activity is a direct result of our in-depth infrastructure review that we have completed on our WC2 cluster.

Update:  As of 1155pm CST, Rackspace Engineers will be starting the maintenance for this evening.  We will continue to update the post as the maintenance progresses. 

Update: As of 1:21AM CST, Rackspace Engineers have completed maintenance for this evening.  If you have any questions, please contact a member of our
support team by visiting us in live-chat or by calling 1.877.934.0407
or INTL +1.210.581.0407.

As of 7:35 AM CST, the Rackspace Cloud system engineers have detected an issue that’s causing degraded performance on our ASP1-5.WC2 cluster in our Dallas / Fort Worth (DFW) data center.  This is an isolated issue impacting this single node only.  We will update this post as the status changes.

UPDATE: As of 8:30 AM CST, system engineers continue to address the performance issues affecting ASP1-5.  We will post here with updates as soon as they are available.

UPDATE: As of 9:50 AM CST, Rackspace Cloud Engineers are continuing to work on the issues affecting our ASP1-5 cluster.  As of this update, the issues are still isolated to this cluster. 

UPDATE: As of 10:30 AM CST, the team remains fully engaged in correcting this issue.  Some sites in our ASP1-3 cluster may also be affected at this time.  We will continue to keep you updated via this post.  

UPDATE: As of 11:00 AM CST, the team has made progress, but they are continuing to actively address the issue.  The performance degradation affecting ASP1-3 has significantly improved and sites should be loading properly.  We will continue to address the ASP1-5 cluster.

UPDATE: As of 11:10 AM CST, at this time, all sites hosted on ASP1-5 should now be loading properly.  The team has continued to monitor the cluster over the past half-an-hour and site performance has remained consistent.  Once again, we appreciate your patience as we corrected this issue.