Outage impacting web experience and pipelines starting

What to do

• Check whether you use CircleCI Pipelines & Workflows, CircleCI UI. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 19m it lasted.

  Anything that failed between 2026-07-02 15:25 UTC and 2026-07-02 16:44 UTC may need a replay.

• Subscribe to CircleCI's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=circleci works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from CircleCI 6

1. investigatingJul 2, 2026, 3:25 PM UTC

   We are currently investigating this issue.

   provider

2. investigatingJul 2, 2026, 4:07 PM UTC

   We are investigating an issue that may cause some customers to be redirected to a login page and prevents some pipelines from starting. Thank you for your patience as we continue our mitigation of this issue.

   provider

3. monitoringJul 2, 2026, 4:24 PM UTC

   We have identified a source of increased system load which caused failures processing API requests. We have mitigated the issue and jobs are now running as expected and UI access has recovered. Some pipelines may have failed or failed to start. Resubmitting should correct these issues. We are continuing to monitor to ensure recovery.

   provider

4. monitoringJul 2, 2026, 4:29 PM UTC

   Pipelines are continuing to run as normal. We will continue monitoring for recovery.

   provider

5. resolvedJul 2, 2026, 4:44 PM UTC

   Pipelines and UI access have fully recovered. Please contact customer support if you have issues running builds. This incident has been resolved.

   provider

6. postmortemJul 10, 2026, 6:57 PM UTC

   ## Summary On July 2, 2026, from 15:12 UTC to 16:44 UTC, some CircleCI customers experienced failures starting pipelines and difficulty accessing the CircleCI UI, including an inability to log in. Customers triggering pipelines via the API may also have had pipelines fail to run. The incident originated from a combination of internal maintenance and customer-initiated project deletions running concurrently, which caused elevated load on a data service. The connection layer between our API and that data service was not configured to time out slow responses under the elevated load conditions seen during this incident. This eventually exhausted available threads and prevented the request routing layer from handling any other requests, resulting in customer-facing errors. The affected services were scaled up and the internal maintenance job was canceled to reduce pressure, allowing all affected functionality to return to normal. Customers whose jobs failed during this window may rerun affected jobs. A small number of customers may still see workflows that appear to be running; this is actively being corrected and does not impact billing. We thank you for your patience while our team worked on implementing a fix. ## What Happened \(all times UTC\) At 13:30 on July 2, 2026, as part of normal platform maintenance, our engineering team began a data deletion job, divided into several hundred individual maintenance tasks. At 14:40, unrelated to our platform maintenance, a significant number of customer project deletion requests were received and began executing. At 15:12, the combined load from the maintenance tasks and the customer-driven projection deletions caused an internal data service to slow down, and this began cascading into our API layer. Because connections between the API layer and the workflows data service were not configured to time out slow responses under elevated load conditions, the API-handling components became overwhelmed, went unresponsive, and began failing their health checks. This caused our request routing layer to lose available upstream requests, resulting in customers seeing errors when attempting to access the UI, to log in via GitHub or Bitbucket OAuth, or to start pipelines. At 15:16, our monitoring systems alerted and engineers were paged. The team investigated and confirmed the source of the elevated load. At 15:30 Engineers began to scale up the affected services and cancelled a long-running maintenance task to drain the load. The team noted signs of that the system was recovering around 16:12 and adjusted the status to monitoring at 16:24. At 16:44, the team confirmed that all customer-facing services had recovered to normal operating levels. Background load from the deletion job drained by approximately 17:00. Following the incident, we discovered a small number of workflows continued to inaccurately report that they were in a running state. This should be corrected within the next few days. In all cases, customers will only be billed for actual compute time used. ## Future Prevention and Process Improvement Our team is implementing the following improvements as a result of this incident: * After the incident was resolved we added timeout protections on connections between our API layer and internal data services, so that slow responses from a downstream service cannot exhaust API worker capacity. * We are actively investigating ways to enforce resource limits on background data tasks to prevent them from impacting customer-facing services. * We are improving observability and alerting on the affected data service so that resource pressure is detected earlier, before it reaches a level that impacts customers.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.