Networking issues

What to do

• Check whether you use Daily API, Core Call Experience. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 38m it lasted.

  Anything that failed between 2024-11-25 16:48 UTC and 2024-11-25 19:26 UTC may need a replay.

• Subscribe to Daily's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=daily works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Daily 7

1. investigatingNov 25, 2024, 4:48 PM UTC

   We're investigating issues that may be causing problems with network connections between regions.

   provider

2. investigatingNov 25, 2024, 5:07 PM UTC

   We're continuing to investigate the source of meeting disruptions. Customers may be experiencing 'meeting moves' where a call session moves from one server to another, causing a 2-3 second disruption to the call. You may also see delays in receiving meeting.started and meeting.ended webhooks.

   provider

3. identifiedNov 25, 2024, 5:10 PM UTC

   We've temporarily disabled the component that sends webhooks.

   provider

4. identifiedNov 25, 2024, 5:25 PM UTC

   We're addressing an issue with an internal database that's causing problems with existing meetings, as well as starting new ones. Your users are likely seeing some failures when trying to join meeting sessions, and users in ongoing sessions are seeing occasional meeting moves. We'll post more information as soon as soon as it's available.

   provider

5. monitoringNov 25, 2024, 5:33 PM UTC

   We've made some changes to the affected database, and our metrics and error rates have returned to normal. We've also re-enabled delivery of all webhooks, and we're monitoring for any further issues.

   provider

6. monitoringNov 25, 2024, 6:08 PM UTC

   Our metrics have stayed at normal levels since our remediating actions about 30 minutes ago. We're continuing to monitor the platform while we discuss longer-term solutions to make absolutely sure we've addressed the root cause here.

   provider

7. resolvedNov 25, 2024, 7:26 PM UTC

   We've deployed an update that increases the throughput of the database that was the bottleneck in today's incident. We'll have more info about additional remediations and a postmortem for today's incident within the next few days.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.