Service updates for popular cloud hosts.
Archive for the ‘Linode’ Category.
Thursday, January 26th, 2012
9:45am (EST): Network maintenance will be performed in Dallas on February 12th between 1:00am and 6:00am (EST). We do not expect any downtime during this maintenance, however a brief period of increased latency and/or packet loss may be observed.
12:14pm (EST): Maintenance will be performed on Atlanta upstream conections on Friday, December 30th, between 12:00am and 1:00am EST. This maintenance will include a brief outage on connections to PCCW/BTN.
Saturday, December 17th, 2011
08:45pm (EST): The network maintenance in the London, UK facility has been completed at this time.
Wednesday, December 14th, 2011
12:45pm (EST): The London, UK facility will undergo brief network maintenance on Saturday, December 17th between 8:00pm and 9:00pm EST (Sunday, December 18th between 1:00am and 2:00am GMT). We do not expect any downtime during this maintenance, however a brief period of increased latency and/or packet loss may be observed.
02:49AM [11/30/2011] (EDT): Connectivity has been stable for the last 20 minutes and we now believe this issue to be resolved. If you're still experiencing issues, please open a support ticket from within the Linode Manager. Your patience and understanding while our upstream provider worked to resolve this issue was greatly appreciated.
02:29AM [11/30/2011] (EDT): Our upstream provider identified the issue to be caused by a route optimization that had been mishandling some prefixes. The issue has now been resolved and connectivity has returned to normal, however we are continuing to monitor at this time.
02:17AM [11/30/2011] (EDT): The Atlanta facility has encountered a networking issue following the scheduled maintenance. Our upstream provider in the Atlanta facility is investigating this issue and we will provide updates as we know more. There is no reason to issue reboot jobs at this time.
3:24pm (EDT): Maintenance will be performed on Atlanta upstream conections on Wednesday, November 30th, between 12:00am and 2:00am EST. Downtime is not anticipated, but some sub-optimal routing may be experienced as traffic is shifted between providers and route optimization updates.
Monday, October 10, 2011
12:13pm (EDT): Network maintenance will be performed in Dallas on Thursday, October 20th between 1:01am and 6:00am EDT. We do not expect any downtime during this maintenance.
2:40pm (EDT): We identified a large-scale Denial of Service attack targeting a Linode in the Newark facility. We have taken the necessary measures to mitigate the attack and connectivity to the Linodes affected by the attack has now stabilised.
2:25pm (EDT): We are aware of a network issue that is affecting some Linodes in our Newark facility. We are investigating and will post an update with more information shortly. There is no reason to issue reboot jobs at this time.
03:15AM (EDT): The network connectivity appears to have stabilized once again. We will continue to monitor the network connectivity. If you are still having connectivity issues please open a support ticket from within the Linode Manager.
02:18AM (EDT): The network upstream of our Fremont facility appears to be experiencing stability issues once again at this time. We’ve alerted the network operation center at the Fremont facility to have them investigate. We’ll provide more information once it is available.
12:21AM [10/4/2011] (EDT): We have received an RFO from Hurricane Electric regarding the incident that occurred this evening:
On October 3rd we experienced a large attack against multiple core routers on a scale and in ways not previously done against us. We had various forms of attack mitigation already in place, we have added more. It was all fixable in the end, just the size and number of routers getting attacked and the figuring out what attacks were doing what to what took some time. The attack mitigation techniques we’ve added will be left in place. We are continuing to add additional layers of security to increase the resiliency of the network.
Because the attackers were changing their methods and watching how their attacks were responded to, we are not at liberty to elaborate on the nature of the security precautions
taken.
11:25PM (EDT): At this time it appears that all routes have stabilized. We are continuing to closely monitor the network stability from multiple locations. If you are still experiencing issues please open a support ticket from within the Linode Manager. Your patience and understanding while this issue was resolved is appreciated.
10:49PM (EDT): There are still some routes that are experiencing increased latency and loss at this time. Network engineers at Hurricane Electric are still working to stabilize the network connectivity as quickly as possible. There is currently no ETA until full resolution. Your continued patience and understanding are greatly appreciated.
08:26PM (EDT): Hurricane Electric’s networking engineers are still working to have full connectivity restored as quickly as possible. We appreciate your continued patience and understanding.
07:32PM (EDT): Reports from Hurricane Electric indicate this was a large scale network-based attack that they have contained at this time. It does appear that routes are starting to come back online. We will provide an update once the network connectivity has been fully restored and has stabilized.
06:07PM (EDT): There appears to be continuing instances of increased latency and losses to the Fremont facility. We are working with the NOC to have this resolved as quickly as possible. We apologize for any inconvenience.
03:15PM (EDT): The following has been received from Hurricane Electric:
The incident was a probable DDOS attack, but its behavior was unusual and difficult to identify. Our network engineers made some adjustments to the DOS countermeasures acquired after last week’s incident, and that seems to have stabilized traffic flow. We apologize for the inconvenience.
-Ben Larsen
Hurricane Electric Internet Services
02:00PM (EDT): The problem has cleared up, and has looked OK for the past 10 minutes or so. We will continue to monitor. We await the RFO from Hurricane Electric.
01:45PM (EDT): The Fremont facility has encountered a networking problem. They are currently investigating and we will update as we know more.
On September 28, 2011 10:20pm PDT and September 29, 2011 11:45am PDT, the Fremont 1 datacenter was subject to a a DDOS targeting a core router. The attack caused OSPF and BGP reloads resulting in elevated CPU utilization and performance degradation of the router.
The incident on September 28, 2011 10:20pm PDT was identified and mitigated at 10:40pm PDT. The incident on September 29, 2011 11:45am was identified and partial mitigation was realized shortly thereafter with full containment at approximately 12:45pm PDT. All systems are fully operational at this time.
We have already been in contact with the router vendor, and have obtained a new software image that addresses this type of infrastructure attack. We will be deploying the new image shortly. A maintenance notification will be sent out separately regarding this emergency maintenance.
Hurricane Electric makes every effort possible to minimize the impact of router hardware and software upgrades, we apologize for the inconvenience.
4:35pm (EDT): Connectivity appears to be stable at this time. We are waiting on information from Hurricane Electric as to the specifics of the issue.
3:50pm (EDT): The Fremont facility is continuing to investigate. Connectivity has returned, but is not stable. This is a network issue and it is not necessary to reboot your Linode. We will provide an update as soon as we have additional information.
3:05pm (EDT): We are investigating a network issue that is affecting a percentage of Linodes in Fremont.
03:05pm (EDT): Our phone system is back online at this time. We apologize for any inconvenience caused by phone support being unavailable.
12:51pm (EDT): We are currently experiencing an issue with incoming calls to the Linode office. We’re working on having this resolved as soon as possible. If you require support and are unable to open a ticket from within the Linode Manager please send an e-mail to supp...@linode.com.
8:35pm (EDT): All Linodes are back online. If you're still experiencing issues, please open a support ticket from within the Linode Manager. Your patience while we worked to resolve this issue was greatly appreciated.
8:05pm (EDT): The majority of affected Linodes are back online. We are still working on bringing the remaining Linodes up and will post an update here shortly.
7:45pm (EDT): There has been a partial power outage in the Fremont facility. Power has been restored to the affected hosts and we are working on bringing Linodes back up.
7:27pm (EDT): Some of our hardware in the Fremont facility has stopped responding. We are investigating this issue now and will post an update with more information as soon as possible.
