CloudFail.net

03:05pm (EDT): Our phone system is back online at this time. We apologize for any inconvenience caused by phone support being unavailable.

12:51pm (EDT): We are currently experiencing an issue with incoming calls to the Linode office. We’re working on having this resolved as soon as possible. If you require support and are unable to open a ticket from within the Linode Manager please send an e-mail to supp...@linode.com.

8:35pm (EDT): All Linodes are back online. If you're still experiencing issues, please open a support ticket from within the Linode Manager. Your patience while we worked to resolve this issue was greatly appreciated.

8:05pm (EDT): The majority of affected Linodes are back online. We are still working on bringing the remaining Linodes up and will post an update here shortly.

7:45pm (EDT): There has been a partial power outage in the Fremont facility. Power has been restored to the affected hosts and we are working on bringing Linodes back up.

7:27pm (EDT): Some of our hardware in the Fremont facility has stopped responding. We are investigating this issue now and will post an update with more information as soon as possible.

Linode apologizes for the downtime in Fremont on Sunday. We do our best to plan for and prevent downtime. Downtime does happen, and just like any hosting provider we are not immune to that fact.

We will be discussing with the datacenter how they plan on preventing issues like this in the future, and continuously checking in with them to ensure they are following through.

Here is the RFO from Hurricane Electric:

At approximately 6:00am PDT, at the Fremont 1 facility, power was interrupted to the datacenter floor when the main UPS input breaker tripped. The breaker was reset, the maintenance bypass closed, and power was restored at approximately 6:20am PDT. Support technicians on site visually inspected customer equipment located in the affected area, restarting any that needed assistance.

UPS technicians will be onsite today to assist us in determining the cause of the failure.

Moving forward, we are now going to replace this UPS system with another brand as soon as possible and we are going to go from an N+1 UPS configuration to a 1+1 electrical system, with each PDU having a static transfer switch to a separate electrical distribution system so that if something happens all the PDUs can be switched over. The time line for this is in the next few months due to the lead time on getting this type of equipment.

Hurricane Electric takes its network and power stability seriously. Please know that we are committed to providing the best service possible.

1:25pm (EDT): All affected Linodes are back online as of 30 minutes ago. If you continue to experience issues please open a support ticket from within the Linode Manager.

11:00am (EDT): Most Linodes have been brought back up (you can check your job queue for a boot job). We are now working on some stragglers. We appreciate your patience.

10:15am (EDT): We are still working through bringing Linodes up and will update here shortly.

9:44am (EDT): There has been a power outage in the Fremont facility. Power has been restored and we are working on bringing Linodes up now.

5:30pm (EST): The routing loop was corrected approximately 30 minutes ago.

4:30pm (EST): We are looking into a network issue that is affecting some routes for a small percentage of Linodes in Atlanta.

Wednesday, July 7, 2011

1:40pm (EDT): Network maintenance will be performed in Atlanta on July 15th at 11:00pm. The window for this maintenance is 2 hours. We do not expect any downtime during this maintenance, however a brief period of increased latency and/or packet loss may be observed.

Monday, June 27, 2011

2:00pm (EDT): The maintenance on this storage pool has been completed and all backups have been successful since early this morning.

Thursday, June 23, 2011

4:00pm (EDT): Over the past several days we have been experiencing problems with one of the storage pools that services the Backup Service in Fremont. Existing data is not at risk, however the problem is affecting the ability of some Linodes in Fremont to back up successfully. We've been working on correcting the problem with minimal downtime, however our efforts have taken longer than originally believed. We have also been discussing our procedures and how we could have handled this situation better.

Unfortunately we need to offline this pool temporarily in order to fully resolve the issue. An update will be posted here once the pool is brought online.

Wednesday, June 22, 2011

11:45am (EST): Network maintenance will be performed in London the night of Wednesday, June 29th between 11:00pm and 3:00am UTC (7:00pm and 11:00pm EDT). We do not expect any downtime during this maintenance.

Friday, June 3, 2011

4:00pm (EDT): The Atlanta facility will undergo network maintenance between 12:00am and 3:00am EDT, June 4th. This maintenance will impact traffic passing through nLayer, causing traffic to reroute through other providers.

12:20pm (EDT): We believe the issue was resolved approximately 20 minutes ago, and we apologize for any inconvenience. We'll continue to closely monitor before marking this issue resolved.

11:40am (EDT): A block of switch ports has gone bad and we are working on correcting the issue as soon as possible.

10:30am (EDT): A routing loop caused a port to become disabled and affected a small percentage of Linodes. The issue has been corrected and connectivity should be back to normal at this time.

10:20am (EDT): We have received reports of a network issue affecting some Linodes in our London facility. We are investigating and will post an update shortly. There is no reason to issue reboot jobs at this time.