Skip to main content
Buildkite

Increased error rates from Test Plan API

Partial outage Lasted 8h 13m · started · Buildkite's page

Level
confirmed
Worst impact
Partial outage
Components
REST API
Down Index
33of 100

What to do

  • Check whether you use Buildkite REST API. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 8h 13m it lasted.

    Anything that failed between 2026-03-10 01:21 UTC and 2026-03-10 09:34 UTC may need a replay.
  • Subscribe to Buildkite's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=buildkite works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Buildkite 3

  1. investigating

    We've observed periodic test splitting plan timing out and falling back to non-intelligent splitting. Performance appears to be back to normal as of an hour ago. We are continuing to investigate the root cause and solve the underlying issue.

    provider

  2. monitoring

    We have implemented several mitigation and continue working on fixing the underlying cause. Our team is actively monitoring the situation to ensure the stability. We will provide further updates as we make progress on resolving this issue.

    provider

  3. resolved

    Our mitigations have resolved the elevated latency and likelihood of suboptimal fallback test plans. We have also identified and fixed a blind-spot in our automated alerting, which was previously unable to detect this scenario as an issue. Work continues this week to resolve the underlying performance issue by restructuring how the relevant data is ingested and accessed.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-8efb8aa59c

Is Buildkite still down?

No, this incident was resolved 2026-03-10 09:34 UTC. For Buildkite's live state, see https://down.guide/buildkite.

How long did it last?

8h 13m, from 2026-03-10 01:21 UTC to 2026-03-10 09:34 UTC, by the times Buildkite posted.

What was affected?

Buildkite named REST API.

What caused it?

Buildkite hasn't published a root cause. If it does, it shows here.

Where does this come from?

Buildkite's official status page, https://www.buildkitestatus.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.