What Went Wrong On January 23, 2025, from 03:43 AM ET to 05:12 AM ET, a planned ISP maintenance event disrupted traffic in a key region, specifically impacting WebSocket connections. Due to temporary routing adjustments made earlier in the week, all WebSocket traffic was directed to this region, creating a single point of dependency. Additionally, the failover mechanism intended to reroute WebSocket traffic to alternate regions did not activate as expected, resulting in a 105-minute service outage.
Immediate Actions: Investigate and resolve the failover mechanism issue to ensure reliable traffic redirection for WebSocket connections during future disruptions.
Future Actions: Enhance routing strategies to prevent dependencies on a single region for WebSocket traffic during maintenance windows. Conduct more comprehensive failover testing to validate system performance under simulated outages.