# Increased latency - US2 cluster - Provider: [Pusher](https://down.guide/providers/pusher) - Confidence: Confirmed - Impact: Degraded - lasted 7h 32m - started 2025-10-19 00:44 UTC - Channels WebSocket client API, Channels presence channels - Pusher states the impact was 2025-10-19 00:44 to 08:17 UTC (7h 33m) - id: DWN-c315e672a9 - Official page: https://stspg.io/cqw0hdd2dqmy ## Updates from Pusher ### 2025-10-19 00:44 UTC: investigating Increased latency and degraded performance on clusters main, us2 and us3. ### 2025-10-19 03:11 UTC: identified The issue has been identified and we are applying mitigations. We are seeing improved latency in main and us3 clusters. us2 still has elevated latency. ### 2025-10-19 04:03 UTC: identified We have scaled all three clusters out to handle the increased traffic. We anticipate the latency to slowly improve. ### 2025-10-19 08:17 UTC: resolved Latency has remained stable over the past three hours. Our investigation identified multiple contributing factors to the earlier latency increase, including network saturation and capacity limits at our cloud providers affecting the US2 cluster. The US3 cluster issue was resolved quickly through autoscaling, while the US2 cluster experienced scaling delays that required manual intervention. We are implementing corrective measures to prevent recurrence and improve system resilience. ### 2025-10-23 14:16 UTC: postmortem Root Cause Analysis: Increased Latency and Message Delivery Failures – US2 Cluster Incident Date: October 19, 2025 Duration: 00:44 UTC – 08:17 UTC Status: Resolved Summary On October 19, 2025, customers using Pusher experienced increased latency and message delivery failures. These issues primarily affected the US2 cluster, with intermittent impact also observed in the MT1 and US3 clusters. The incident resulted in delayed or undelivered messages for many applications. Latency stabilized at 08:17 UTC after mitigation actions were completed. Impact Between 00:44 UTC and 08:17 UTC, multiple customers experienced: \* Delayed or failed message delivery across affected clusters \* Degraded performance in connection establishment and publishing The most significant and prolonged impact occurred in the US2 cluster, while MT1 and US3 clusters saw elevated latency for a shorter period before stabilizing. Root Cause The primary cause of the incident was IP address saturation within the subnet assigned to the public Pusher clusters. When traffic levels increased, the US2 cluster was unable to scale out further because the available IP addresses in its subnet were fully utilized. This IP scaling limitation prevented the creation of additional instances needed to handle the load. Secondary factors included temporary network saturation and capacity limits at our cloud provider, which amplified the latency in the early stages of the incident. Detection and Response The issue was first detected through a combination of customer reports and internal monitoring alerts showing elevated response times and connection errors. The timeline of actions was as follows: \* 00:44 UTC – Monitoring alerted the team to increased latency across MT1, US2, and US3 clusters. \* 03:11 UTC – Engineers identified subnet capacity as a contributing factor; mitigations began. \* 04:03 UTC – All clusters were scaled out to distribute traffic; latency began to improve in MT1 and US3 clusters. \* 08:17 UTC – Manual intervention allowed US2 to scale successfully, restoring normal latency. Resolution To restore service, the engineering team: \* Scaled out MT1 and US3 clusters to handle increased traffic loads \* Monitored all clusters to confirm sustained stability Once additional capacity was provisioned and high loads normalized, latency levels returned to normal and remained stable. Preventative Actions To prevent recurrence, Pusher has initiated the following actions: \* Rate Limits: We will re-evaluate how rate limits are implemented to better mitigate content from neighboring customers on the shared clusters. \* Subnet Expansion: Re-evaluating and increasing the size of subnets assigned to shared clusters to ensure sufficient IP availability for future scaling events. \* Load Balancer Enhancements: We will implement load balance sharding in order to better distribute connections. \* Capacity Planning Improvements: Enhancing internal monitoring and alerting for subnet and IP utilization thresholds. Next Steps and Commitment We recognize that message latency and delivery reliability are critical to our customers’ applications. Our team is continuing a full review of cluster capacity management and provider configuration to improve resilience under high traffic conditions. We apologize for the disruption this incident caused and appreciate your patience while we worked to resolve it. Ensuring reliability and transparency remains our highest priority. From the builder: does your app depend on Pusher? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=pusher) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Pusher. --- down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-c315e672a9), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.