The CrossBrowserTesting team was alerted at approximately 4:15AM Central time (9:15AM UTC) to a heavily degraded performance state. Upon investigation, it was discovered that a redundant piece of network infrastructure had failed over correctly, but devices had not handled the failover gracefully. We resolved the issue with the devices and tested the platform to verify.
At approximately 8:25AM Central time (1:25PM UTC) we were alerted to a slowdown in the platform that was causing sessions to take significant lengths of time to start up after requests. An investigation into the cause pointed to the previous failover causing a previously-unseen issue in our network management services, leading to significant delays in certain operations.
The issue has now been resolved, and we will be implementing next steps to ensure such a failure state does not cause further issue in the platform.
Posted Oct 22, 2019 - 09:48 CDT
This incident affected: Selenium Test and Screenshot Test.