Few customers experiencing intermittent errors during pipeline execution.

What to do

• Check whether you use Harness Continuous Delivery (CD) - FirstGen - EOS, Continuous Integration Enterprise(CIE) - Linux Cloud Builds. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 25m it lasted.

  Anything that failed between 2026-07-09 20:21 UTC and 2026-07-09 21:46 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 4

1. investigatingJul 9, 2026, 8:21 PM UTC

   We are currently investigating this issue.

   provider

2. monitoringJul 9, 2026, 9:10 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. resolvedJul 9, 2026, 9:46 PM UTC

   This incident has been resolved.

   provider

4. postmortemJul 21, 2026, 5:21 AM UTC

   # Summary On July 9, 2026, some customers in the Harness Prod3 cluster experienced pipeline failures over a ~2.5-hour window, despite making no changes in their pipelines. Shell Script steps failed when fetching scripts from the Harness File Store, and Kubernetes deployment steps failed when retrieving secret encryption details. # Impact 1. **Affected users**: Some customers using the Harness Prod3 cluster reported pipeline execution failures. 2. **User impact:** Customers experienced failures in Shell Script and Kubernetes deployment steps. Impacted pipelines required manual retries after the incident was resolved. 3. **Scope:** This was not a platform-wide outage. The issue was isolated to specific APIs in our internal service within the Prod3 cluster. # Root Cause The incident was caused by degraded performance in a downstream service. This led to a storm of timeout errors in our internal service, which caused these failures. # Mitigation Engineering identified the change that caused this issue in the downstream dependency service and deployed a patch to restore it. Once deployed, pipeline executions returned to normal success rates. # Preventive Measures & Next Steps 1. Added additional monitoring and alerting on the downstream service to detect performance degradation earlier. 2. Reviewing timeout and retry logic in the internal service to improve resilience against downstream latency spikes. 3. Evaluating circuit breaker patterns to prevent cascading failures from downstream dependencies.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.