Intermittent webhook delivery failures

What to do

• Check whether your product depends on LiveKit Cloud. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2026-07-30 22:55 UTC and 2026-07-30 22:55 UTC may need a replay.

• Subscribe to LiveKit Cloud's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=livekit works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from LiveKit Cloud 2

1. resolvedJul 30, 2026, 10:55 PM UTC

   Between 2026-07-16 and 2026-07-29, a caching bug caused webhooks to be intermittently not delivered for about one hour for projects whose configuration was updated during this period.

   provider

2. postmortemJul 30, 2026, 10:56 PM UTC

   ## Summary Between 2026-07-16 and 2026-07-29, a caching bug caused webhooks to be intermittently not delivered for about one hour for projects whose configuration was updated during this period. ## Root Cause A bug in an internal configuration service caused webhook settings to be dropped from a project's cached configuration whenever that project's record was updated. Once triggered, our system stopped delivering webhooks until the cache refreshed from the database, up to one hour later. Because of this caching behavior, the impact was intermittent: webhooks would stop, then resume on their own, then stop again after the next project update. Additionally, the "test webhook" feature in the dashboard reads webhook configuration through a different path and was unaffected, so test webhooks succeeded even while real event delivery was failing. ## Timeline \(UTC\) * 2026-07-16 23:48 - Change containing the bug reached production; intermittent impact begins * 2026-07-24 - Some customer reports of missing webhooks received; investigation begins * 2026-07-28 16:23 - Root cause identified; fix developed and rollout begins * 2026-07-29 16:26 - Fix deployed to all regions; incident resolved ## Scope of Impact Customers using webhooks were affected across all regions between 2026-07-16 and 2026-07-29, with most impact occurring after 2026-07-22. A project was affected only if its configuration was updated during the window, and each occurrence lasted up to one hour. Affected projects saw `room_started` and `participant_*` webhooks silently not delivered, while `room_finished` webhooks and dashboard webhook tests often continued to work. If you observed missing webhooks and had recently made any settings changes to your project, you were likely impacted. ## Mitigations and Follow-ups * The bug has been fixed and a regression test has been added. * We added logging of webhook delivery response codes to improve visibility into delivery failures. * We are adding per-project webhook delivery monitoring and alerting that can detect missing deliveries independent of overall volume. * We are unifying the configuration paths used by webhook delivery and the dashboard webhook test, so a passing test reflects actual delivery behavior. * We are making the configuration cache lifetime adjustable so similar issues can be mitigated quickly in the future.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.