Service updates for popular cloud hosts.
Archive for 2nd February 2010
Rackspace Cloud engineers are performing a emergency maintenance on our Control Panel. The total duration of the maintenance will be approximately 10 minutes. During this time the control panel will be unavailable. We expect the maintenance to be quick and will include updates of our progress as they happen in real time.
UPDATE: As of 7:50 PM CST, The Rackspace Cloud engineers have completed the maintenance. If you have any questions, please contact a member of our support team via live-chat or at the following telephone numbers: 24-hour toll free 1.877.934.0407 and INTL +1.210.581.0407
As part of our ongoing efforts to monitor our user base for odd activity, we noticed a sudden surge in followers for a couple accounts in the last five days. Given the circumstances surrounding this, we felt it was best to push out a password reset to accounts that were following these suspicious users.
Then we started doing some digging and, given what we found today, we felt it important to share this information. An outline of what appears to have happened follows …
Torrent sites aren’t exactly “new”; however, this is one of the first times that we’ve seen an attack that came from this vector. It appears that for a number of years, a person has been creating torrent sites that require a login and password as well as creating forums set up for torrent site usage and then selling these purportedly well-crafted sites and forums to other people innocently looking to start a download site of their very own. However, these sites came with a little extra — security exploits and backdoors throughout the system. This person then waited for the forums and sites to get popular and then used those exploits to get access to the username, email address, and password of every person who had signed up. Additional exploits to gain admin root on forums that weren’t created by this person also appear to have been utilized; in some instances, the exploit involved redirecting attempts to access the forums to another site that would request log-in information. This information was then used to attempt to gain access to third party sites like Twitter. We haven’t identified all of the forums involved (nor is it likely that we’ll be able to, since we don’t have any connection with them), but as a general rule, if you’ve signed up for a torrent forum or torrent site built by a third party, you should probably change your password there.
The takeaway from this is that people are continuing to use the same email address and password (or a variant) on multiple sites. Through our discussions with affected users, we’ve discovered a high correlation between folks who have used third party forums and download sites and folks who were on our list of possibly affected accounts. While not all users who were sent a password reset request fall into this category, we felt that it was important to put this knowledge out there so that users would know of the possibility of compromise of their data by a third party unrelated to their Twitter account. We strongly suggest that you use different passwords for each service you sign up for; more information on how to keep your Twitter account safe can be found here: http://twitter.zendesk.com/forums/10711/entries/76036.
Del Harvey
Director, Trust and Safety
As of 5:20 PM CST, The Rackspace Cloud engineers have identified issues that’s causing degradation in our Target 02 Storage Array in SAT 1 (San Antonio, TX) Data Center. We’re working diligently to solve this issue and will provide updates to this blog once we’ve received more information. For further updates please refer to this post as our support channels may be full at this time.
UPDATE: As of 5:52 PM CST, The Rackspace Cloud engineers are continuing to investigate the issue causing degradation in this storage node. We’re continuing to work toward resolution and will send another update once we get more information.
UPDATE: As of 6:20 PM CST, The Rackspace Cloud engineers have corrected the issues in our SAT1 Target 02 Cluster. We apologize for the inconvenience that you may have experienced during this time.
Update(11am PST): A fix has been issued and this should be considered resolved.
As of 11:00am CST, the Rackspace Cloud Systems engineers have temporarily
disabled CRON. Any CRON jobs you
may have scheduled will not run. As soon as our System Engineers have
applied the corrective actions being worked on now we will re-enable
and update this post. Currently, there is no ETA on when this will be
corrected but we’re working diligently to resolve. Please check this post for any news and updates.
Update: As of 5:00pm CST Cron has been re-enabled for DFW.WC2. If you have any other questions, please do not hesitate to contact a
member of our support team via live-chat or at the following
telephone numbers: 24-hour toll free 1.877.934.0407 and INTL
+1.210.581.0407.
On February 4, The Rackspace Cloud system administrators will be performing priority maintenance on two of our MySQL database clusters in our DFW data center.
Maintenance will be performed on MySQL50-46.wc1 from 2:00 am – 3:00 am CST.
Maintenance will be performed on MySQL50-02.wc1 from 3:00 am – 4:00 am CST.
During the maintenance window some SQL queries utilizing these clusters will be in a read locked state (INSERT, UPDATE, DELETE, ALTER queries will be ignored) for the duration of the maintenance. There may be a period of up to 10 minutes in which databases will be unavailable.
We expect this maintenance to last no longer than the time listed. We apologize for the inconvenience, but we need to complete this work in a timely manner. Please note that no other technologies will be effected by this maintenance.
We have identified the cause of the DNS issue and applied a fix. DNS propagation lag is now the only component that is causing any continuing problems but customers have started to report that their DNS is now working again. We are still tracking any remaining issues so please watch this space or feel free to monitor updates in Slicehost chat.
The DNS tool in Slice Manager is now working. We are currently investigating other DNS-related issues. If you are experiencing any problems, please watch this space or feel free to monitor updates in Slicehost chat.
