CloudFail.net

We are experiencing a network outage for many of our sites hosted in the Herakles Data Center, such as those on EY01 and EY05.  Please watch your e-mail for our automated updates.

We have resolved the communication outage on EY01.  This outage was between the cluster and the internet, and as such, no actual slices were impacted.  All sites are now available again.

We are experiencing an outage on EY01, which started at 6:38 PDT.  Please watch your e-mail for our automated updates.

We had an issue with our primary and secondary gateways flapping and causing some brief outages. We have restored network functionality on the primary gateway but have shutdown the load balancer service on the secondary gateway to stop the flapping. We are investigating the issue and will restore full redundancy after we resolve the flapping issue.

We are currently experiencing gateway issues on the EY04 cluster.  Our engineers are working to resolve the issue.

Network engineering have re-established full network connectivity and redundancy. Some network components were replaced and to ensure network stability, they will be further tested tonight at 3am CST. A full investigation and post-mortem will be done and the results released on request to our xCloud customers after 3 days.

Network engineering has implemented a short-term solution to re-enable xCloud connectivity, and traffic is flowing normally. We are still pursuing root cause, and may need to interrupt service later today in order to replace components that are still suspect. Both our infrastructure partner as well as the networking vendor are on-site and actively engaged on this problem, we’ll continue to send updates as we make progress.

The intermittent network connectivity still persists. Our xCloud infrastructure partner is fully engaged and have senior engineers from their network hardware vendor on site to resolve this issue. They are working on a solution to restore full network functionality.

The xCloud network is still having intermittent connectivity issues. Along with our infrastructure partner, our engineers continue to work on the connectivity problem to restore full network functionality.