Observability upload and Agent Simulations failures

What to do

• Check whether you use LiveKit Cloud Cloud Dashboard (cloud.livekit.io) in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 3h 43m it lasted.

  Anything that failed between 2026-10-13 10:01 UTC and 2026-10-13 13:44 UTC may need a replay.

• Subscribe to LiveKit Cloud's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=livekit works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Root cause

We have identified the cause of the Agent Insights issue and are working on a fix.

Updates from LiveKit Cloud 6

1. investigatingOct 13, 2026, 10:01 AM UTC

   We are investigating an issue with Agent Insights where agents are unable to upload their observability data at the end of their sessions. Some recent agent sessions may be missing from the dashboard's sessions list or shown as “In progress”. Agent sessions themselves are not affected. We will post an update shortly.

   provider

2. identifiedOct 13, 2026, 10:45 AM UTC

   We have identified the cause of the Agent Insights issue and are working on a fix.

   provider

3. identifiedOct 13, 2026, 11:40 AM UTC

   We are continuing to work on a fix for this issue.

   provider

4. identifiedOct 13, 2026, 1:15 PM UTC

   Agent Insights uploads are now operational and session data has been storing normally since 12:40 UTC. We are still investigating an issue with Simulations. Simulation runs may complete without a transcript, summary, or verdict. We will post an update shortly.

   provider

5. resolvedOct 13, 2026, 1:44 PM UTC

   This incident has been resolved. Between 08:15 and 12:35 UTC, agents were not able to upload agent observability data to LiveKit's servers in all regions. Agent sessions completed normally, but, unfortunately, any Agent Insights data which attempted to upload within that window (traces/transcripts/logs/recordings) is unrecoverable. The ability to download existing Agent Insights data was impaired until 13:15 UTC. Agent Simulations were unavailable due to a upstream dependency and came back online at 13:25 UTC. Sincere apologies for the impact. We will follow up with a postmortem as soon as possible.

   provider

6. postmortemOct 21, 2026, 11:40 PM UTC

   **Root Cause** During a security rotation of access credentials, we revoked a storage credential that we believed was no longer in use. It was still being used by our observability platform, which stores the data behind the dashboard and the Analytics API, the Agent Insights data that agents upload at the end of each session, and by the service that serves that data back for downloads and Agent Simulations. Once the credential was revoked, uploads failed in all regions. Restoring service took longer than a credential swap normally would for three reasons: * New credentials had to be set up, and each affected service had to be updated to use them. * The new credentials took longer than expected to become available in all regions \(about two hours in our US East region\). * Once the observability platform was working again, we found additional services that also needed updating. The incident was resolved once they were updated. **Timeline \(UTC\)** * 2026-10-13 08:13 - A storage credential is revoked during a security rotation. Observability uploads begin failing in all regions. * 08:15 - Agent Insights transcript and audio downloads and Agent Simulations begin failing. * 09:39 - Our team detects the failures and opens an incident. * 10:45 - The revoked credential is identified as the cause. * 11:16 - Affected services are reconfigured to use a new credential. * 12:40 - The new credential takes effect in the last region. Observability uploads recover in all regions. * 13:13 - Agent Insights downloads recover. * 13:26 - Agent Simulations recover. * 13:30 - Incident resolved. * 2026-10-17 - Recovery of observability data from the impact window completes \(see below\). **Scope of Impact** * **Observability uploads \(08:13 to 12:40 UTC, all regions\):** Observability data written during this window, including the data behind the dashboard and the Analytics API, failed to store. Nearly all agent sessions that ended in this window also failed to upload their Agent Insights data, so in Agent Insights these sessions were missing from the sessions list, or shown as "In progress". Agent sessions themselves ran normally, and media and calls were not affected. * **Data recovery:** In our resolved update we said all Agent Insights data from this window was unrecoverable. We have since restored most of it. **Session recordings and transcripts from this window are permanently lost.** All other observability data for these sessions has been restored and is visible in the dashboard and through the Analytics API. * **Agent Insights downloads \(08:15 to 13:13 UTC\):** Downloading transcripts and audio for existing sessions failed during the incident window. * **Agent Simulations \(08:15 to 13:26 UTC\):** Simulation runs couldn't load conversation transcripts, so they finished without a summary or verdict, or were cancelled. Runs from this window need to be re-run. * **Billing:** Agent Insights usage is only recorded after a successful upload, so failed uploads were **not** billed. **Detection and Response** We detected this issue about 85 minutes after it began. We did not have alerting on upload failures to observability storage, so it was caught when an engineer noticed elevated errors rather than by an automated alert. We also initially marked the incident resolved once uploads recovered, before confirming that Agent Simulations and downloads had recovered too. **Mitigations and Follow-ups** * We have added alerting that pages our on-call team when observability storage uploads fail, along with a continuous check that stored data can still be read and downloaded and an end-to-end test that verifies our storage credentials still work. * We have built tooling to replay and restore observability data from a past window, which we used to recover data from this incident. * We are moving these services to dedicated service credentials managed with our infrastructure and auditing other services for the same pattern. * We have updated our credential rotation process to verify every consumer of a credential before revoking it, and to account for regional propagation delays when rolling out replacements. * We are adding redundancy so a single failed credential can no longer stop observability uploads. We sincerely apologize for the disruption. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.