The root cause if the outage was due to cpu and memory resource contention that caused the orchestration engine to become unresponsive. We have mitigated further occurrences by adding more infrastructure to handle the load; increasing the number of servers processing orchestration and increasing memory and processing resources allocated to the servers. Stability of MXO has improved with these enhancements and our engineers are continuing to closely monitor the systems to ensure continued stability.
Posted Apr 30, 2024 - 17:37 EDT
Resolved
This incident has been resolved.
Posted Apr 23, 2024 - 12:36 EDT
Update
We are continuing to monitor for any further issues.
Posted Apr 23, 2024 - 09:54 EDT
Monitoring
We have stabilized the issue in the European production environment as of 2:18 CET and are continuing to monitor it.
Posted Apr 23, 2024 - 09:54 EDT
Investigating
We are currently experiencing a major outage in our European production environment as of 12:01 PM CET. Our engineers are currently investigating the issue. We will send an additional update in 60 minutes or when more information is available.