We have implemented configuration changes and updated our database infrastructure over the weekend, and full stability has been restored to our European production environment (as of December 1st, 4:15 CET). We will provide a postmortem with more details about the root cause of the issue during this coming week.
Posted Dec 01, 2019 - 23:52 EST
Update
Customers are currently experiencing degraded performance on our European production environment (as of 09:30 UTC on November 29th, 2019). We are taking measures to restore availability and performance and as such there will be short periods of increased API & Interaction error rates over the next few hours. We will provide further updates during the day and when more information becomes available.
Posted Nov 29, 2019 - 08:27 EST
Update
Customers are currently experiencing degraded performance on our European production environment (as of 09:30 UTC on November 29th, 2019). We will provide further updates during the day and when more information becomes available.
Posted Nov 29, 2019 - 06:26 EST
Update
The European production environment has remained stable throughout the night. We continue to have slower than normal response times and are continuing to actively monitor it.
Posted Nov 28, 2019 - 09:10 EST
Monitoring
We have implemented a temporary configuration change in our database servers and performance of the European production environment has stabilized. In the meantime, we are continuing to work with vendors to find the root cause of this incident and will continue to provide updates until the issue is resolved.
Posted Nov 27, 2019 - 18:02 EST
Update
We currently have a ticket open with our database vendor to address the issue and they are actively investigating our nodes to help identify where the issue lies. In addition, they are also reviewing our configuration files to provide feedback.
We have also reached out to our cloud provider's critical support team to help investigate if the issue lies within our hosted cloud platform since the issue is isolated to a subset of servers.
Posted Nov 27, 2019 - 14:28 EST
Identified
Customers may be experiencing high response times on our European production environment (as of 11:35 UTC on November 27th, 2019). Our engineers have identified the cause of the disruption as an issue with some of our database nodes and are working to resolve. We will provide updates during the day.