CloudFail.net

We are currently investigating increased error rates in the East Coast facilities.

Between 1:45 PST and 2:28 PST, some customers experienced problems loading the Amazon S3 Console. This issue has been resolved and the console is operating normally.

Some customers are currently unable to load the Amazon S3 Console. We are actively working to resolve the issue.

We just wanted to provide a little more information on the List-Buckets API issue yesterday morning. From 1:39AM PDT to 5:39 AM PDT yesterday morning, calls to the List-Buckets API returned incomplete results. The List-Buckets API calls completed successfully, but in many cases, the results only included a subset of the customers buckets. This caused the AWS Management Console to show an incomplete list of buckets or indicate there were no buckets at all. Note that during this event all Amazon S3 buckets were still online, calls to GET, PUT, and DELETE were operating normally, and no customer data was lost.

The cause of this event was a software update. Last week wed deployed a software update to a subset of data centers in each region as part of a staged deployment. Over the last week we encountered no issues. Monday night, we completed the world wide deployment. This deployment included a change to the internal mechanism S3 uses to list bucket metadata. This internal mechanism is used in many places including the process that synchronizes metadata between our primary bucket index and the secondary index that supports the List-Buckets API. Last night we encountered an unexpected error condition that caused the internal mechanism for listing bucket metadata to return a partial set of results to the synchronization process. The synchronization process updated the secondary index based on the partial results and incorrectly removed the buckets that were not included. Subsequent calls to the List-Buckets API then returned incomplete results. We fixed the incomplete results issue by repopulating the secondary index, which was completed at 5:39 AM PDT on Tuesday. At this point, the List-Buckets API was returning the complete results and we began rolling back the defective software update worldwide. This roll back was completed at 7:00AM PDT.

We tested this change before we began the deployment, but clearly missed this case. We wont deploy another change until weve extended our testing and alarming to ensure weve covered this and related cases. We will also change our synchronization process to halt and alarm if it appears that theres been a large, sudden decrease in the number of buckets, as this is a highly unusual case. We apologize for any inconvenience or confusion that resulted from this event.

Sincerely,


The Amazon S3 team

Between 10:29 AM PDT and 3:11PM PDT we experienced increased connection timeouts to our West coast facilities in the US-Standard region. The issue has been resolved and the service is operating normally.

We have confirmed that a subset of requests routed to our West coast facilities in the US-Standard region are experiencing connection timeouts. We are working on resolving the issue.

We are currently investigating reports of increased connection timeouts for requests that are routed to our West coast facilities.

Between 9:00AM PST and 10:55AM PST, all requests for CreateBucket and DeleteBucket returned errors. Latency for a small percentage of GET, PUT, and LIST requests was also elevated. The issue has been resolved and the service is operating normally.

All requests to create new buckets are currently returning errors. We are also experiencing increased latency for a small percentage of requests.

Correction Time of events are from 9.09pm 10.49pm PDT.