The root cause if the outage was due to cpu and memory resource contention that caused the orchestration engine to become unresponsive. We have mitigated further occurrences by adding more infrastructure to handle the load; increasing the number of servers processing orchestration and increasing memory and processing resources allocated to the servers. Stability of MXO has improved with these enhancements and our engineers are continuing to closely monitor the systems to ensure continued stability.
Posted Apr 30, 2024 - 17:38 EDT
Resolved
The issue has now been resolved as of 11:10 am CET. Further details will be provided in a post-mortem report within 48 hours.
Posted Apr 25, 2024 - 08:52 EDT
Monitoring
We have stabilized the issue in the European production environment as of 11:10 AM CET and are continuing to monitor.
Posted Apr 25, 2024 - 06:30 EDT
Investigating
We are currently experiencing a major outage in our European production environment as of 9:35 AM CET. Our engineers are currently investigating the issue. We will send an additional update in 60 minutes or when more information is available.