CloudFail.net

Starting around 5:20p Pacific we starting seeing an unusually high number of errors on the site as well as increased latency. We are working on this problem now.

Update (5:45p): This problem has been resolved.

UPDATE (10:50 CST): Appears to have been resolved. Thanks to all for your feedback and contacting your ISPs.

UPDATE (9:23 CST): This appears to also affect AT&T’s residential customers in the area. If you are affected by this issue, we advise you to contact your ISP to help contribute to a speedy resolution.

UPDATE (8:00pm CST): Our networking team has identified that the issue appears to be related to a link between AT&T and Level3. Affected customers are using AT&T/SBC Business Class DSL and are advised to contact their ISP to help contribute to a speedy resolution.

UPDATE (7:11pm CST): Our networking team is still looking into this and we will update this post as soon as they have more information.

Some California customers are currently reporting routing issues from Cogent and are unable to access Slices in STL-B.

We are looking into it and will let you know when we have more information.

At 12:03am PDT we experienced an unusually high amount of traffic hitting our Palo Alto datacenter causing reachability issues for our users in Australia and areas near Palo Alto. We immediately failed over to our datacenters in Los Angeles and Seattle and DNS services recovered almost immediately. Our website, however, is primarily hosted out of Palo Alto and took another 30 minutes or so before we were able to appropriately filter the traffic down. All services are currently operational but we’re leaving DNS down in Palo Alto for the night while we continue to investigate. Seattle and Los Angeles will handle all DNS for the west coast until further notice.

UPDATE: We’ve brought DNS services back up in Palo Alto. We’ve also taken steps to eliminate the abnormal traffic we saw last night, which should prevent this in the future.