CloudFail.net

We are experiencing a network issue in our Miami-DC. We are having intermittent network connectivity and packet loss.

We are working on it right now, we will update this post as soon as more information is available.

UPDATE 6:15PM EST: We are still having issues with connectivity, we have all hands on deck working on the problem. More updates following.

UPDATE 6:30PM EST: We are still working on this issue. Please hang on tight. We are working as fast as we can to get this resolved.

UPDATE 7:00PM EST: The network is now up. All Webbies are also up, and did not go down. We are are getting more information in detail so we can update this ticket.

UPDATE 7:55PM EST: The initial report is that we experienced an massive inbound DoS service attack, which seems to have been related to our primary uplink provider Cogent as a massive packet flood to our network. During this attack our core Cisco routers also locked up due to an un-fixed module bug by Cisco and prevented our main uplink to failover to our secondary which is Level3. After we changed everything to Level3 we came back up again which makes us believe that the packet flood was an issue created directly by Cogent and caused a Denial of Service to us. We are now on Level3 as a we wait for a report from Cogent and we will update this blog post when that information becomes available.

As of 12:30AM EST we are investigating some server lockups that have cause the intermittent uptime. At this time we are investigating whether we had a RAID controller issue and we had to reboot the server. As of this time the server is up and running and operational, however we are now doing more investigations to solve the kernel panics we are receiving. If the fix doesn’t solve the problem we will be swapping the ram on the server even though the server reports no RAM errors.

If the server continues to give us more issues, we will then schedule a migration of every single customer to a new server.

Note: As of 12:30 AM EST All Webbies are up and running. We will update the Status post as we perform any reboots or changes.

Update: 3PM EST – We will be scheduling a memory swap on this server shortly. We will post the schedule here.

We are performing some maintenance on Wash-03 at the moment. Should be back  up shortly. More to follow.

## UPDATE ##

18:56 EST

All webbies are now back online after a system reboot.

## UPDATE ##

19:08 EST

If anyone is experiencing any problems, please submit a support ticket.

We are currently working through some network issues in our Washington DC datacenter. We are experience intermittent connectivity and some packet loss.

More updates to follow as we find out what’s happening from our network engineers.

** UPDATE **

DC is reporting DoS attack, they are currently mitigating and the network should be stable back soon.

We had received a large DoS attack towards the Wash-01 datacenter that caused increased latency and packetloss for some customer servers beginning 8:40AM CDT (13h40 UTC) 2011-MAY-02. The victim IPs were null routed and the issue mitigated by 8:57AM CDT (13h57 UTC) 2011-MAY-02 and all services are returned to normal at this time.

Hello,

We experienced a hardware issue on our raid on Wash-03, the server is up and running now. However we are investigating the matter closely, we will be updating this post as more information becomes available.

4:03AM EST – We’re investigating an issue affecting users in miami-b14. Please hang on while we find an issue, we’ll update this blog post asap.

6:00 AM EST – We had a critical raid failure. We are on recovery mode and Webbies in miami-b14 are currently down. We are migrating each Webby’s data so that and starting them on other nodes.

7:30 AM EST – Webbies still being migrated.

4:00 PM EST - The majority of Webbies have been migrated. We have 5-6 Webbies till being migrated manually in order to ensure data integrity.

7:00 PM EST – The majority of Webbies were recovered yesterday. We had to do hard data recovery for the last few ones. We are working on the last 2 at this point.

4:00 AM – EST – All Webbies have been recovered. If your Webby has any issues still please contact support.

We’re working on an issue with the Webby Manager. We’ll update this post as soon as we get updates.

Note: All Webbies are up.

Update: The Webby Manager is back online, please send us a ticket if you run into any issues.

Thanks.

Some customers in miami-b07 node are experiencing connectivity issues. We’re investigating and will post updates soon.

Update 2:30PM EST: All webbies on miami-b07 are up and running. Problem resolved.

[12:00 PM EST] We are experiencing connectivity issues with the Miami location and it’s also affecting connectivity between Webby Manager and Dallas data center.

We’re working on mitigating this issue, please stand by.

12:15 PM EST Connectivity has been resumed and all services are back up.

[09:45 PM EST] We are experiencing connectivity issues with the 67.23.79.x IP Range and it’s also affecting connectivity between Webby Manager and Dallas data center.

We have full hands on deck working on a fix.

Please standby for updates.

[09:55 PM EST] Connectivity has been restored.