The incident was related to a caching layer becoming exhausted before scaling could take effect. Our engineers mitigated the issue by replacing the affected cache and we are reviewing our alerting policies to catch the effects of the issue sooner.
Posted Apr 08, 2021 - 08:36 EDT
Resolved
This incident has been resolved. Our engineers have applied a mitigation and the North American Production environment has remained stable. Further details will be provided in a post mortem report within 48 hours.
Posted Apr 05, 2021 - 20:53 EDT
Update
Our engineers have identified the root cause of the issue and are currently applying a mitigation. The North American production environment has remained stable and we are continuing to monitor.
Posted Apr 05, 2021 - 20:40 EDT
Monitoring
Our North American production environment experienced a partial outage between 8:06PM to 8:17PM UTC. Customers may have experienced heightened response times or occasional 500 errors. We are monitoring the environment as we continue to investigate the root cause.