Skip to main content
Orb

Elevated errors across API endpoints

Partial outage Lasted 1h 8m · started · Orb's page

Level
confirmed
Worst impact
Partial outage
Components
Reads, Ingest, Writes, Analytics
Down Index
39of 100

What to do

  • Check whether you use Orb Reads, Ingest, Writes, Analytics. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 1h 8m it lasted.

    Anything that failed between 2025-09-03 13:28 UTC and 2025-09-03 14:36 UTC may need a replay.
  • Subscribe to Orb's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=orb works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Orb 18

  1. investigating

    We are currently investigating database issues causing API errors

    provider

  2. investigating

    We are continuing to investigate this issue.

    provider

  3. investigating

    We're continuing to investigate, and are actively pursuing mitigation strategies. We apologize for the disruption and will provide status updates diligently here as we learn more.

    provider

  4. investigating

    We are continuing to see elevated errors on write endpoints. We believe we understand the root cause, and are pursuing multiple parallel mitigation strategies to resolve the incident as quickly as possible.

    provider

  5. investigating

    Although this incident is still active, we're seeing partial recovery for specific customers. We're continuing to treat this as top priority and working to mitigate the impact by running maintenance operations at our database layer.

    provider

  6. investigating

    We're seeing persistent partial recovery across writes, but a lower rate of failures persist. We believe the mitigations we've put in place are helping, but are continuing to pursue faster and more encompassing mitigation strategies. Note that the ingestion API is not failing and has not during the incident; once the incident is resolved we do not expect any data gaps with event ingestion so no retries should be necessary.

    provider

  7. investigating

    We're continuing to focus on mitigating impact. Once again, we apologize for the disruption and will publish an RCA after the incident is resolved.

    provider

  8. investigating

    We are continuing to investigate the issue.

    provider

  9. monitoring

    We're seeing broader recovery, and have seen no recent API errors since 12:38 UTC and now working to bring back async services.

    provider

  10. monitoring

    Continuing to see broader recovery - API errors have recovered.

    provider

  11. monitoring

    We're continuing to see broad recovery, and there have been no API errors since 12:38 UTC. Continuing to work to bring back async services.

    provider

  12. monitoring

    Our services are continuing to recover, including issuing any outstanding invoices and webhooks. We have not seen any elevated rate of API errors since initial recovery.

    provider

  13. monitoring

    We are continuing to work through our asynchronous work queue.

    provider

  14. monitoring

    We are continuing to work through our asynchronous work queue.

    provider

  15. monitoring

    We are nearly recovered, but will keep the incident in monitoring state until all asynchronous work is fully stable.

    provider

  16. monitoring

    We are continuing to work through our asynchronous work queue.

    provider

  17. monitoring

    We are nearing full resolution, and will continue to keep this incident updated.

    provider

  18. resolved

    This incident has been resolved. We'll continue to monitor for any disruptions, and follow up with a detailed RCA.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-2b8ecf285b

Is Orb still down?

No, this incident was resolved 2025-09-03 14:36 UTC. For Orb's live state, see https://down.guide/orb.

How long did it last?

1h 8m, from 2025-09-03 13:28 UTC to 2025-09-03 14:36 UTC, by the times Orb posted.

What was affected?

Orb named Reads, Ingest, Writes, Analytics.

What caused it?

Orb hasn't published a root cause. If it does, it shows here.

Where does this come from?

Orb's official status page, https://status.withorb.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.