CloudFail.net

Connectivity to all impacted instances has been restored. The service is now operating normally.

We are starting to restore connectivity to the impacted instances. We will post an update as soon as the connectivity for all instances is restored.

We are investigating the connectivity issue. We will post more details as soon as we have an update.

We are investigating connectivity issues for a small number of RDS database instances in a single Availability Zone in the us-east-1 region.

We want to provide some additional information on the Internet connectivity interruption that impacted our US-East Region last night. A networking router bug caused a defective route to the Internet to be advertised within the network. This resulted in a 22 minute Internet connectivity interruption for instances in the region. During this time, connectivity between instances in the region and to other AWS services was not interrupted. Given the extensive experience that we have running this router in this configuration, we know this bug is rare and unlikely to reoccur. That said, we have identified and are in the process of deploying a mitigation that will prevent a reoccurrence of this bug from affecting network connectivity.

We understand that when networking events affect instances in multiple Availability Zones it causes our customers serious operational issues that are difficult to architect around. We have been using and refining our Availability Zone architecture for over 10 years at Amazon to provide highly reliable services. Availability Zones provide a high degree of isolation including physical separation, independent power distribution, independent cooling and mechanical systems, and multiple physical links to the Internet through multiple transit providers and peering connections. All of our regions have exceeded 99.99% availability over the last several years. We are also continually investing in improving our architecture as we learn more. In addition to the remediation discussed above which addresses the specific bug we saw last night, we are currently in the later stages of refining the way that we do route advertisement within a region. These changes will isolate any bad route information to inside a single Availability Zone while maintaining the performance characteristics of our current inter-Availability Zone network design. We have been deploying these changes carefully to avoid impact to customers, but we expect these changes to be complete within the next several weeks. We are confident these changes will protect us from multi-Availability Zone impact for the sort of bug we saw last night.

Amazon FPS will be performing scheduled maintenance from 02:30 AM to 03:00 AM Pacific Time on 20th March 2012.
During the maintenance period you may experience elevated error rates impacting CBUI, ASP, and FPS API calls in the NA region.

Between 2:22 AM and 2:43 AM PST, some instances in Virtual Private Cloud experienced an outage of Domain Name Service (DNS) resolution. This issue is corrected and the service is operating normally.

Between 2:22 AM and 2:43 AM PDT, some instances in Virtual Private Cloud experienced an outage of Domain Name Service (DNS) resolution. This issue is corrected and the service is operating normally.

Between 2:22 AM and 2:43 AM PST internet connectivity was impaired in the US-EAST-1 region. Full connectivity has been restored. The service is operating normally.