CloudFail.net

SoftLayerNotify: 18-May-2012 14:45 UTC – Network Engineers shifted traffic off the troubled link to alleviate private network issues to/from SNG01 [588]

SoftLayerNotify: 18-May-2012 12:25 UTC Customer’s servers may experience brief periods of packet loss on the backend network to and from the SNG01. [586]

Emergency Power Maintenance [05/18/2012]
Date: Friday, May 18, 2012 (05/18/2012)
Start Time: 09:00 AM CDT
End Time: 12:00 PM CDT
Services affected: Power
Location: DAL05
Duration: 3 hours

Hello,

This ticket is to inform you of an upcoming required EMERGENCY maintenance.

To avoid any lengthy downtime we will be migrating your CCI(s) to another host. This will be done prior to the power maintenance. The migrations will start at 0900 CDT.

Our technicians will be on site to ensure that the server has the minimum downtime possible.

We do apologize for any inconvenience that this may cause and the short notice. We have identified a problem with one of the Input breakers facilitating power to your server(s). Based on the results of a diagnostics performed by the vendor, we have made the decision to swap this out immediately to prevent an unplanned outage event.

Please let us know if you have any questions.

18 May 2012 07:47am (GMT): We are having major network stability issue affecting all our SLC clouds at the moment.  Onsite Engineers are working on addressing the issue currently.

18 May 2012 09:20 (GMT):  UPDATE  The networks at the SLC datacenter are more stable now, but the issue is not fully resolved.  All clouds except for SLC-F are currently working, and our L3 Engineers are working to finish the repair on all these clouds.

18 MAY 2012 09:45 (GMT): UPDATE  The network issues have been resolved.

May 17 2012 10:17PM We are experiencing intermittent availability issues with Windows Azure Service Bus in the North Europe sub region. We are actively investigating this issue and working to resolve it as soon as possible. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.
May 17 2012 11:02PM We have confirmed that the Service Bus availability issues in North Europe were related to the intermittent connectivity issues with SQL Azure in North Europe. Our monitoring shows that Service Bus availability is substantially restored in this sub region. Please look to the SQL Azure feed for North Europe, for status updates on the SQL Azure intermittent connectivity issue. We apologize for any inconvenience this causes our customers.
May 18 2012 12:17AM The intermittent connectivity issues with SQL Azure in North Europe have been resolved. Service Bus availability has been fully restored and verified in this sub region. We apologize for any inconvenience this caused our customers.

May 17 2012 8:17PM A small number of users are experiencing intermittent connectivity issues in SQL Azure. We are actively investigating this issue and working to resolve it as soon as possible. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.
May 17 2012 9:57PM We are still troubleshooting this issue and verifying the most probable cause. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

May 17 2012 8:17PM A small number of users are experiencing intermittent connectivity issues in SQL Azure. We are actively investigating this issue and working to resolve it as soon as possible. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.
May 17 2012 9:57PM We are still troubleshooting this issue and verifying the most probable cause. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers.

Switch Replacement Maintenance [05/25/2012]
Date: Friday, May 25, 2012 (05/25/2012)
Start Time: 12:01 AM CDT
End Time: 02:01 AM CDT
Services affected: Private Network Connectivity
Location: Houston02, Server Room 2, Rack 98, All Slots
Duration: 2 hours

Scheduled Maintenance – 5024257 – bcs98a.sr02.hou02 – Houston
Date: Friday, May 25, 2012 (05/25/2012)
Start Time: 05:01 AM UTC
End Time: 07:01 AM UTC
Services affected: Private Network Connectivity
Location: HOU02
Duration: 2 hours

==================================================
Network Engineers have identified an issue on the rack-level switch, bcs98a.sr02.hou02, which provides private network
connectivity for your server(s). To prevent an unplanned outage resulting in significant downtime, Engineers
will be replacing this switch during a scheduled maintenance window.

Private network services including the following (but not limited to) will be impacted:

- Server to server connectivity on the private network
- iSCSI
- DNS
- NAS
- eVault
- PPTP / SSL VPN connectivity to impacted servers

While the maintenance window is set for 2 hours, we expect 30 minutes of downtime.

Again, NO downtime is expected on the public network.
==================================================

If you have any problems after this time frame with regard to connectivity, or if you have any questions
regarding the maintenance at any point, please open a new ticket in the customer portal referencing this
ticket. This ticket is not being tracked by personnel and will be only used to update status of this maintenance.

We appreciate your patience during this work and welcome any feedback.

Thank you,
Network Engineering

4:21pm EDT/8:21pm GMT: We’re investigating now.

UPDATE: 4:40pm EDT/8:40pm GMT: And we’re back!