Elevated API errors and delayed async processing

What to do

• Check whether you use Orb Reads, Writes, Analytics. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 16h 34m it lasted.

  Anything that failed between 2024-10-29 13:09 UTC and 2024-10-30 05:44 UTC may need a replay.

• Subscribe to Orb's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=orb works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Orb 4

1. investigatingOct 29, 2024, 1:09 PM UTC

   A query load issue has been identified starting at 2024/10/29 8:45 UTC which was causing intermittent spikes of errors accessing internal data storage, resulting in API errors and delays in async processing. This issue has been mitigated for the majority of customers as we continue to investigate.

   provider

2. monitoringOct 29, 2024, 2:12 PM UTC

   API services stabilized at 12:30 UTC and async processing has been restored, and we are continuing to monitor our systems.

   provider

3. monitoringOct 29, 2024, 7:39 PM UTC

   API services and asynchronous processes are stable. The query load issue has been fully isolated and we're working on improvements to prevent this issue in the future.

   provider

4. resolvedOct 30, 2024, 5:44 AM UTC

   This incident has been resolved.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.