CloudFail.net

Another of our databases crashed at 11:30 am PST. The crash was the result of the same bug encountered on 2/24, but was a different database instance. With assistance from Percona we inspected the core file to determine the corrupt table which was then dumped and dropped while in recovery mode and then restored with [...]

Around 21:21 Pacific on 02/24/2012 one of our shards crashed. We were able to bring the  database back online at 23:39 PM. During the outage customers on that shard were unable to access their data. We also were not collecting data on that shard during that time. It seems like we hit MySQL bug 61104. [...]

We will be doing some upgrades to our database shards on Friday March 9th starting around 8PM Pacific time.  There will be an outage on each shard as it is upgraded. We anticipate no more than 30 minutes downtime per shard. During the maintenance customers will be unable to post or view data. We will [...]

Yesterday evening around 5:30 PM Pacific we experienced an outage caused by a fiber cut at our co-location site. In theory the fiber cut should have had no effect as it was a redundant loop.  In practice it triggered a routing event that caused a complete outage for at least 15 minutes. Our co-location provider [...]

For 35 minutes yesterday (Jan 31) between 4:25 and 5:00pm PST we experienced sporadic problems with our service. The problems effected everything — our Real User Monitoring beacon lost some data during that time; and while the collection tier handling application agent data was also impacted that is likely invisible to our customers because of [...]

We apologize to our customers for any inconvenience. Filed under: Service Updates

We are seeing an issue with 1 of our collectors that is causing problems saving data for some customers. Root cause: We added some disks to the array on one of our databases. We had done this twice on test systems and once on another production system with little visible impact. This particular database server [...]

Some customers are unable to access rpm. Updates ASAP Filed under: Service Updates

One of our DBs will be off line for a short while. Some customers will be unable to access their RPM data. Filed under: Service Updates

We are moving to our new Chicago data center tonight. The site will be down for several hours. 7:07pm – Site taken down 7:18pm – Broke replication between EY and Chicago 7:30pm – Snapshots of Chicago databases taken 7:45pm – Sanity checked Chicago databases 7:55pm – Start proxying agent traffic from EY to Chicago 8:10pm [...]