October 10 MyMPS Outage

To the students, teachers, and administrators of Minneapolis Public Schools

We’d like to take a moment to apologize for the recent itslearning system outage and the problems it caused. While it’s no consolation, we want to provide a more detailed explanation of the events that transpired and the actions taken in response.

At 2:31 pm ET Monday October 10th, services in our primary hosting center went down following a hardware failure. Normally such an error would not cause any disruption, but the fail over mechanism did not function properly. This resulted in system-wide downtime for customers hosted out of our Oslo data center and degraded performance for customers hosted in our US data center. A complete root cause analysis will be made available once investigations have been completed.

The system was restored at 2:50 am ET for our US hosting environment and 3:30 am EDT for our Oslo based environment. For three hours following the point of restore some customers experienced issues with notification and real-time messaging refreshes (messages could be sent and delivered).

At 1:00 pm ET Tuesday October 11th, after working diligently with the hosting center and hardware provider, it was determined that the best course of action was to take the system off line in order to perform major server maintenance to permanently fix the problem.

At 9:30 pm EDT Tuesday, the system was taken down and faulty hardware was replaced. The downtime lasted 4 hours and exceeded the communicated time of 3 hours.

At 1:30 am EDT Wednesday October 12th, the system, its new servers and enhanced fail-over system having been tested and certified, was restored.

We share our customers’ frustration and disappointment over this incident. Please know that we are committed to providing the world’s best and most reliable learning platform. We will do everything in our power to ensure a similar situation never happens again.

Next steps include:

  • A full root cause failure analysis, the results of which will be shared with our customers
  • A rigorous enhancement of our internal processes in order to improve the timeliness of critical customer communications
  • A critical review with our hardware provider to ensure compliance with our Service Level Agreements, and to prevent future problems

We greatly appreciate the 7 million global students, teachers, parents, and administrators that rely on our solutions every day. We sincerely pledge to earn and maintain your trust and surpass your expectations in everything we do.

Thank you,

Arne Bergby

CEO, itslearning