Disconnections and failed connections in multiple regions

What to do

• Check whether you use LiveKit Cloud US East - Ingress, US West - Ingress, US West - Real Time Communication, US Central - Real Time Communication and 7 more in ca, eu, me, us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2026-10-31 07:43 UTC and 2026-10-31 07:43 UTC may need a replay.

• Subscribe to LiveKit Cloud's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=livekit works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from LiveKit Cloud 2

1. resolvedOct 31, 2026, 7:43 AM UTC

   This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected and continued their conversations, but a small number of agent sessions ended or restarted. Some SIP call transfers also failed. - About 1.3% of Room Service API requests failed during this period. ListParticipants and GetParticipant were the most affected. Some WHIP ingress update and delete requests also failed. The bug was triggered when a specific type of data message was relayed between servers hosting the same multi-region room. We stopped the trigger at 07:43 UTC, and there were no further restarts after that. A fix was deployed to all regions by 08:49 UTC. Apologies for the impact. We will follow up with a postmortem sharing more details on impact and RCA as soon as possible.

   provider

2. postmortemNov 3, 2026, 1:54 PM UTC

   **Summary** On October 31, 2026, between 07:10 and 07:43 UTC, an uncommon data pattern surfaced a latent software bug which caused media servers in several regions to restart repeatedly. A small percentage of participants were disconnected and the vast majority automatically reconnected. Impact was concentrated in North America. We stopped the trigger at 07:43 UTC and deployed a fix to all regions by 08:49 UTC. We apologize for the disruption this caused. **Root Cause** Applications can open their own data channel and send raw data over it, outside of LiveKit's standard data message format. Media servers forward this raw data to other participants in the room, wrapping it with the sender's identity so that receivers know who sent it. When a room spans multiple regions, the media servers hosting it also forward this data to each other. That server-to-server step sent the raw data without the sender's identity, and the receiving server assumed the identity would be present, which caused it to crash. This gap in our server-to-server protocol had been present for some time and was not introduced by a recent deployment. Starting at 07:10 UTC, one application sent raw data over its own data channel in a room that spanned multiple regions. Every server that received the forwarded message crashed at the same time, while the server the sender was connected to kept running and kept sending. Participants on the crashed servers automatically reconnected to other servers. Those servers then began hosting the room, received the next message, and crashed as well. This cycle repeated until we stopped the trigger at 07:43 UTC. In total, media servers in 12 of our data centers restarted, some of them up to 10 times. **Timeline \(UTC\)** 2026-10-31 07:10 - An application begins sending raw data over its own data channel in a multi-region room. Media servers in US East and Canada restart within seconds. 07:13 - Our monitoring detects the restarts and an incident is opened automatically. 07:10 to 07:43 - Media servers in 12 data centers restart repeatedly as participants reconnect to new servers. 07:43 - We block the traffic triggering the bug. There are no further restarts after this point, and customer impact ends. 07:46 - A fix for the bug is ready. 08:04 - The fix begins rolling out, starting with a small set of regions. 08:49 - The fix is deployed to all regions. **Scope of Impact** Realtime sessions: * Across all regions, about 1.5% were disconnected more than once within five minutes. * In US East, US Central, US West, Canada and Mexico, about 6% were disconnected more than once within five minutes. * The median time from disconnect to rejoin was about 3 seconds. Agents: * Agents were disconnected and reconnected the same way as other participants, with the same median reconnect time of about 3 seconds. An agent that reconnects keeps its conversation state, so most conversations continued after a short gap. Fewer than 1% of active agent sessions did not recover. Ingress, Egress and SIP: * Most active WHIP ingress streams were interrupted, and reconnects typically took about 40 seconds. Some WHIP ingress update and delete requests also failed. * About 5% of active egress sessions were disconnected. Most reconnected within a few seconds. * About 1% of SIP participants were disconnected, and half of them rejoined in under a second. Some SIP call transfers also failed. APIs: * About 1.3% of Room Service API requests failed during the window. ListParticipants and GetParticipant were the most affected. How to tell if you were affected: participants in affected sessions saw a reconnecting state, or a brief loss of audio and video. In session analytics, an affected participant appears to leave and rejoin the same room with the same identity between 07:10 and 07:43 UTC. **Detection and Response** Our monitoring detected the server restarts about 3 minutes after they began, and an incident was opened automatically. Identifying the specific room and traffic triggering the restarts took about 30 minutes. Once we found it, we blocked the trigger at 07:43 UTC and the restarts stopped immediately. We deployed a fix to all regions by 08:49 UTC. We did not post to our status page while the incident was in progress, and published a notice only after it was resolved. Our original post mistakenly sent notifications as we initially over-estimated the scope of impact. **Mitigations and Follow-ups** * We fixed the bug. Media servers now handle forwarded data that arrives without the sender's identity instead of crashing. This fix was deployed to all regions by 08:49 UTC. * Raw data forwarded between media servers now includes the sender's identity, so receiving servers always know who sent it. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.