CloudFail.net » newrelic

Service Problems

cloudfailbot — Wed, 01 Feb 2012 17:37:36 +0000

For 35 minutes yesterday (Jan 31) between 4:25 and 5:00pm PST we experienced sporadic problems with our service. The problems effected everything — our Real User Monitoring beacon lost some data during that time; and while the collection tier handling application agent data was also impacted that is likely invisible to our customers because of [...]

Site issue has been resolved

cloudfailbot — Wed, 17 Aug 2011 03:29:46 +0000

We apologize to our customers for any inconvenience. Filed under: Service Updates

Site issue 08/15-08/16 2011

cloudfailbot — Tue, 16 Aug 2011 16:30:27 +0000

We are seeing an issue with 1 of our collectors that is causing problems saving data for some customers. Root cause: We added some disks to the array on one of our databases. We had done this twice on test systems and once on another production system with little visible impact. This particular database server [...]

Having trouble with a database

cloudfailbot — Fri, 18 Feb 2011 06:03:27 +0000

Some customers are unable to access rpm. Updates ASAP Filed under: Service Updates

Going to do some DB maintenance

cloudfailbot — Sat, 12 Feb 2011 05:30:40 +0000

One of our DBs will be off line for a short while. Some customers will be unable to access their RPM data. Filed under: Service Updates

Site down for scheduled maintenance

cloudfailbot — Sat, 06 Nov 2010 02:07:46 +0000

We are moving to our new Chicago data center tonight. The site will be down for several hours. 7:07pm – Site taken down 7:18pm – Broke replication between EY and Chicago 7:30pm – Snapshots of Chicago databases taken 7:45pm – Sanity checked Chicago databases 7:55pm – Start proxying agent traffic from EY to Chicago 8:10pm [...]

Reminder: RPM is down due to a scheduled 6 hour maintenance window.

cloudfailbot — Sat, 06 Nov 2010 02:05:36 +0000

We will notify you when service is restored. Filed under: Service Updates

Reminder: RPM maintenance window today Nov. 5 at 19:00 Pacific Time

cloudfailbot — Sat, 06 Nov 2010 01:15:26 +0000

Reminder: Maintenance window today Nov. 5 at 19:00 Pacific Time. Due to our datacenter migration, RPM service will be temporarily interrupted today November 5, from 7 pm PST/10 pm EST to November 6th, 1 am PST/ 4 am EST. Service will resume as soon as the switch is made and the system is thoroughly tested. [...]

Site Problems

cloudfailbot — Thu, 04 Nov 2010 15:32:42 +0000

At 7:30am we started getting alerts from our site. Upon investigating we learned that one of our databases was hung. The hang was caused by a bad disk volume. Our technicians determined that we had suffered a double disk failure and that the volume was hosed. Update 8:32am – We are bring up a replica [...]

Site Outage

cloudfailbot — Wed, 27 Oct 2010 13:25:08 +0000

At 4:30am our site went down due to a physical server that crashed. The server was running two of our databases, which account for about 20% of customer data. The crashed server initially took down the entire site, as our UI was hung up on the missing databases. At 5:30am we untangled that mess and [...]