CloudFail.net

Our primary carrier on the west coast (NTT) is suffering a major degradation of service that is causing packet loss between our San Jose, Palo Alto and other west coast facilities. This is also causing the website to have intermittent slowness and database connection timeout issues. We have shutdown all circuits on the west coast with NTT and all appears to be stabilizing.

From approximately 00:00 to 04:00 UTC on 9/30/11 we will be performing routine maintenance on our routing equipment at our Washington DC facility. During this time frame all services will be re-routed to alternate facilities. No service impact is expected by this maintenance and all services will continue to operate as normal.  Users who are normally routed to our Washington DC facility will automatically be re-routed to the closest alternate facility.

UPDATE: 03:44 UTC – Our maintenance has been completed and Washington DC is back online serving traffic as normal. No services were impacted by this maintenance.

At approximately 08:22 AM UTC we had a database error occur which resulted in users not being able to login to their accounts and multiple websites being flagged as phishing sites, even though they weren’t. We have since corrected the database error and are looking into the cause at this time.

We are planning on performing system maintenance at 2300 UTC (7pm Eastern) on Wednesday September 14th.  All traffic to Washington DC will be re-routed to an alternate facility during this maintenance.

UPDATE 2300 UTC 9-14-2011: This planned maintenance is commencing now

UPDATE 0228 UTC 9-15-2011: Maintenance is complete all services are restored in Washington DC

At approximately 10:45pm Pacific 9/8/11. Many of our users experienced false phishing alerts when browsing certain sites. A database glitch caused the whitelist to be ignored for a short period. We are very sorry this happened and are in the process of ensuring this never occurs again. We’ll provide more details in the near future.

Router maintenance in LON.  All traffic for this site will be re-routed to an alternate site.

UPDATE: This has been completed.  Traffic has been restored.

Scheduled maintenance in Dallas.  All traffic will be re-routed to an alternate site.   ETR 30 mins.Update: This is complete.  Traffic has been restored to normal.

We are currently carrying out scheduled maintenance at our Miami facility.  All traffic is being re-routed to an alternate facility during the maintenance.

UPDATE:  This maintenance is now complete.

We are currently carrying out scheduled maintenance at our Los Angeles facility.  All traffic is being re-routed to an alternate facility during the maintenance.

UPDATE:  This maintenance is now complete.

Router maintenance in DFW.  All traffic for this site will be re-routed to an alternate site.UPDATE:  Maintenance complete.  All traffic is being routed normally.