Calls failing due to worker unavailability

What to do

• Check whether your product depends on Vapi. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 6d 4h it lasted.

  Anything that failed between 2025-12-17 19:06 UTC and 2025-12-23 23:18 UTC may need a replay.

• Subscribe to Vapi's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=vapi works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Vapi 3

1. updateDec 17, 2025, 7:06 PM UTC

   We have detected an issue with our call workers not scaling to meet demand. The team is investigating and will update here.

   provider

2. updateDec 17, 2025, 7:20 PM UTC

   The system has recovered. The team is monitoring, and we will update here with a full RCA.

   provider

3. resolvedDec 23, 2025, 11:18 PM UTC

   # [IR] Dec 17th — Call Worker Degradation — Object Storage Upload Errors ## Summary On December 17, 2025, at 10:25 AM PST, we observed degradation in our Call Worker service. The issue was caused by Call Workers becoming blocked while uploading call recordings to a downstream object storage provider facing an outage of their own. The incident was fully resolved by 11:02 AM PST, once the downstream provider recovered and Call Workers returned to normal operation. ## Timeline (PST) - **10:25 AM** — Initial call degradation alert triggered - **10:40 AM** — Investigation began - **10:45 AM** — Downstream provider partially recovered - **10:52 AM** — Downstream provider fully recovered - **11:02 AM** — Call Workers fully recovered ## Root Cause A downstream object storage provider experienced a partial outage, during which call recording upload requests began failing or stalling. Requests either timed out or returned 502 errors. These stalled upload operations increased processing time within Call Workers, leading to worker exhaustion. Due to this resource saturation, the system was unable to scale quickly enough to accept new incoming calls, resulting in dropped or unaccepted calls during the affected period. ## Impact For approximately 30 minutes, a subset of calls could not be accepted or were dropped due to unavailable or terminated Call Workers. There was no data loss. **Calls not picked up (worker not available):** - Daily organizations: 15,555 - Weekly organizations: 478 ## What Went Well - Autoscaling eventually resolved the issue of workers being unavailable without the need for manual intervention. ## What Went Poorly - No fallback mechanism was in place for object storage uploads. - Monitoring did not quickly identify the downstream dependency as the root cause. ## Remediation - [ ] Make the object storage upload process asynchronous - [ ] Add more aggressive timeouts and retries for upload operations - [x] Investigate procedures for manually scaling capacity during incidents - [x] Add monitoring for object storage upload errors --- If working on realtime distributed systems excites you, consider applying: https://jobs.ashbyhq.com/vapi/295f5269-1bb5-4740-81fa-9716adc32ad5

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.