Degraded SIP Performance in US West

What to do

• Check whether you use LiveKit Cloud US West - SIP in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 26m it lasted.

  Anything that failed between 2026-10-16 16:04 UTC and 2026-10-16 17:30 UTC may need a replay.

• Subscribe to LiveKit Cloud's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=livekit works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from LiveKit Cloud 6

1. investigatingOct 16, 2026, 4:04 PM UTC

   We are investigating degraded SIP performance in US West starting at 14:49 UTC affecting new outbound calls.

   provider

2. identifiedOct 16, 2026, 4:12 PM UTC

   We have identified the source of the issue and are working to route around the impact.

   provider

3. monitoringOct 16, 2026, 4:35 PM UTC

   We applied mitigation as of 15:58 UTC and SIP outbound connection success has returned to normal. We are continuing to monitor before marking this resolved.

   provider

4. monitoringOct 16, 2026, 4:57 PM UTC

   We estimate that approximately 14% of outbound calls and 5% of inbound calls in US West failed between 14:49 UTC and 15:58 UTC. We will follow up as soon as possible with more details.

   provider

5. resolvedOct 16, 2026, 5:30 PM UTC

   This incident has been resolved. Between 14:49 UTC and 15:58 UTC, approximately 14% of outbound calls and 5% of inbound calls in US West failed. We will follow up with more details surrounding the root cause and preventative actions in a postmortem.

   provider

6. postmortemOct 22, 2026, 2:22 PM UTC

   **Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affected SIP server started with an IPv6 address 2026-10-16 ~15:48 - Customers began reporting failures to our support team 2026-10-16 15:58 - SIP traffic began moving away from the final affected cluster due to automatic cluster draining 2026-10-16 15:59 - Last affected call; customer impact ended 2026-10-16 17:08 - Began rolling out the long term fix requiring the usage of IPv4 2026-10-16 ~21:28 - Fix rollout completed 2026-10-16 ~22:30 - SIP traffic restored to the affected cluster **Scope of Impact** Between 14:49 and 15:59 UTC on 2026-10-16, SIP calls handled by the two affected servers in US West failed. During that window, about 14% of outbound calls in US West were rejected at setup and about 5% of inbound calls were answered and then dropped without audio. **Detection and Response** Our customers noticed this before our monitoring did. Our alerts watch for error-level logs, but carrier rejections are logged as routine events since they happen during normal operation. We also weren't tracking call failure rates on individual servers so our on-call engineers were paged in manually about an hour after the first failure. **Mitigations and Follow-ups** * We deployed a fix so SIP servers only use IPv4 addresses when determining their public address. * We'll add safeguards and automated tests to ensure a SIP server's call setup messages only contain IPv4 addresses. * We are additionally adding improvements in monitoring so that we can proactively detect and prevent these sorts of failures. We apologize for the disruption this caused. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.