Web Calls and Phone Calls Disruption

What to do

• Check whether you use Retell AI Web Call, Phone Call. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 9m it lasted.

  Anything that failed between 2026-09-05 23:10 UTC and 2026-09-06 00:19 UTC may need a replay.

• Subscribe to Retell AI's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=retell works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Retell AI 3

1. investigatingSep 5, 2026, 11:10 PM UTC

   Starting at 4:10 PM, web and outbound calls began failing, while inbound calls experienced connection delays. We are actively investigating the issue and will provide an update as soon as more information is available.

   provider

2. resolvedSep 6, 2026, 12:19 AM UTC

   The issue affecting web, outbound, and inbound calls has been resolved. All calling services are operating normally.

   provider

3. postmortemSep 6, 2026, 3:35 AM UTC

   ### Summary On September 5, a rare race condition in our custom LLM call path caused call scheduling queues across our fleet to stall. Because the failure also made containers appear underutilized, autoscaling never triggered. Between roughly 10:00 and 16:00 PST the impact grew until most containers were stuck and new calls could not be placed. ### Impact * **Web calls, outbound calls, and inbound calls** were all affected. * Inbound calls were the least affected, since they hold the highest priority in our scheduling queue. * Peak impact occurred around 16:00 PST. ### Root Cause The failure required a specific sequence of events: 1. **Stalled initialization.** If a call ended before its custom LLM websocket was established, the call's initialization never completed. The scheduling queue waited indefinitely for that initialization to finish. 2. **Queue exhaustion.** Each container allows a fixed number of concurrent dequeue operations. Enough stalled initializations consumed all available slots, leaving the container unable to pick up new calls. 3. **Misleading scaling signals.** The call-end path still ran and decremented the container's concurrency counter. As a result, stuck containers reported low concurrency and low resource utilization — so no scale-out was triggered. With every container blocked and autoscaling idle, new calls had nowhere to go. This pattern is only reachable through that exact ordering, which is why it went undetected until it occurred at scale. ### Timeline \(PST\) * 10AM: A customer's custom LLM calls begin dropping before initialization completes. * Afternoon: The customer scales up traffic; the number of stalled initializations grows. * ~16:00: Nearly all containers are blocked on dequeue slots. Autoscaling does not trigger. Call placement is broadly impacted. * After 16:00: Mitigation begins: manual scale-out of containers and clearing of stuck concurrency for the affected customer. Service recovers. ### Prevention 1. \(COMPLETED\): Fix the race condition in custom LLM call initialization 2. \(IN PROGRESS\): Add monitoring and alerting on call scheduling queues to detect stuck states 3. \(IN PROGRESS\): Decouple the scheduling queue from call initialization so a slow or failed init cannot block dequeue 4. \(IN PROGRESS\): Add call scheduling queue pressure as an input to our autoscaling policies ### Closing We apologize for the disruption. We are treating this as an opportunity to harden our scheduling and scaling infrastructure, and we will ship the remaining action items within a few days.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.