Skip to main content
Harness

Slowness in Prod1 and Prod2 environment

Degraded Lasted 6h 38m · started · Harness's page

Level
confirmed
Worst impact
Degraded
Components
Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery - Next Generation (CDNG), Continuous Delivery - Next Generation (CDNG), Cloud Cost Management (CCM), Cloud Cost Management (CCM), Continuous Error Tracking (CET), Continuous Error Tracking (CET), Chaos Engineering, Chaos Engineering, Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Integration Enterprise(CIE) - Windows Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Custom Dashboards, Feature Flags (FF), Feature Flags (FF), Security Testing Orchestration (STO), Service Reliability Management (SRM), Internal Developer Portal (IDP), Infrastructure as Code Management (IaCM), Software Supply Chain Assurance (SSCA), Software Engineering Insights (SEI) / AI DLC Insights, Code Repository, Artifact Registry, Platform, AI Test Automation, FME, FME
Down Index
25of 100

What to do

  • Check whether you use Harness Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery - Next Generation (CDNG), Continuous Delivery - Next Generation (CDNG) and 25 more. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 6h 38m it lasted.

    Anything that failed between 2026-09-08 21:52 UTC and 2026-09-09 04:30 UTC may need a replay.
  • Subscribe to Harness's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 10

  1. investigating

    We are currently investigating this issue.

    provider

  2. investigating

    We are continuing to investigate this issue.

    provider

  3. investigating

    We are continuing to investigate the issue.

    provider

  4. investigating

    We are continuing to investigate this issue.

    provider

  5. identified

    The issue has been identified and a fix is being implemented.

    provider

  6. identified

    We are continuing to work on a fix for this issue.

    provider

  7. monitoring

    A fix has been implemented and we are monitoring the results.

    provider

  8. monitoring

    A fix has been implemented and we are monitoring the results.

    provider

  9. resolved

    This incident has been resolved.

    provider

  10. postmortem

    ### Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period.  ### Customer Impact * Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. * No data loss or corruption occurred. ### Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. ### Mitigation * Disabled the capability across all environments * Temporarily increased platform capacity to restore stability ### Next Steps To prevent recurrence, Harness will: 1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query 2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-95ddabb6a0

Is Harness still down?

No, this incident was resolved 2026-09-09 04:30 UTC. For Harness's live state, see https://down.guide/harness.

How long did it last?

6h 38m, from 2026-09-08 21:52 UTC to 2026-09-09 04:30 UTC, by the times Harness posted.

What was affected?

Harness named Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery - Next Generation (CDNG), Continuous Delivery - Next Generation (CDNG) and 25 more.

What caused it?

Harness hasn't published a root cause. If it does, it shows here.

Where does this come from?

Harness's official status page, https://status.harness.io, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.