Skip to main content
Buildkite

AWS us-east-1 single availability zone outage

Degraded Lasted 1d 3h · started · Buildkite's page

Level
confirmed
Worst impact
Degraded
Regions
us, us-east-1
Down Index
31of 100

What to do

  • Check whether you use Buildkite in us, us-east-1. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 1d 3h it lasted.

    Anything that failed between 2026-05-08 01:12 UTC and 2026-05-09 04:34 UTC may need a replay.
  • Subscribe to Buildkite's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=buildkite works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Buildkite 11

  1. investigating

    We're aware that AWS is reporting availability zone failures in us-east-1. We are monitoring the situation but so far there is no customer impact.

    provider

  2. investigating

    A small subset of our customers are experiencing delayed notifications. We are actively provisioning additional capacity for these customers. Availability zone automatic failovers are occurring in response to the outage, and this is causing some brief error blips for some customers.

    provider

  3. investigating

    We are continuing to actively monitor the impacts of this availability zone outage for Buildkite customers. Some transient errors are visible due to availability zone failover events.

    provider

  4. investigating

    We have provisioned additional capacity in unaffected availability zones so that they are able to support the additional load. Automatic failovers continue to occur where necessary. Some latency and transient errors will be visible to customers.

    provider

  5. investigating

    We are actively moving resources out of us-east-1c. Similar brief latency and error blips will be visible to customers while these manual failovers occur.

    provider

  6. investigating

    We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips will unfortunately continue while these manual failovers occur.

    provider

  7. investigating

    We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur.

    provider

  8. investigating

    We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur. (Apologies if you receive duplicated notifications for this update.)

    provider

  9. monitoring

    Despite the ongoing AWS incident, our own services are now stable. We are continuing to monitor our services closely, and are ready for further action should the need arise. We are also watching AWS services closely as they recover.

    provider

  10. resolved

    The upstream AWS incident in us-east-1 has been resolved by AWS, and all Buildkite services are operating normally. No further customer impact is expected. We appreciate your patience during this incident.

    provider

  11. postmortem

    ## Service impact On 8th May 2026 UTC, between 00:00 and 07:30 UTC, some customers would have seen intermittent errors and latency spikes across many areas of the platform. ## Incident Summary The AWS availability zone incident in `use1-az4` triggered our automatic availability failover mechanisms on database and cache clusters, as per AZ-failure tolerant design. During the failover we saw some isolated request errors that were handled by client-side retries in the agent. Customer workloads were either entirely undisrupted or in the worst case saw elevated latency for a period of up to 5 minutes. Throughout the incident we monitored customer impact and prepared additional resources in a healthy availability zone to manually failover to if the automated systems proved insufficient. These were not necessary, and all our infrastructure self healed.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-18c1ab1027

Is Buildkite still down?

No, this incident was resolved 2026-05-09 04:34 UTC. For Buildkite's live state, see https://down.guide/buildkite.

How long did it last?

1d 3h, from 2026-05-08 01:12 UTC to 2026-05-09 04:34 UTC, by the times Buildkite posted.

What was affected?

Buildkite named in us, us-east-1.

What caused it?

Buildkite hasn't published a root cause. If it does, it shows here.

Where does this come from?

Buildkite's official status page, https://www.buildkitestatus.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.