Degraded performance on prod4

What to do

• Check whether you use Harness Continuous Delivery - Next Generation (CDNG), Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Integration Enterprise(CIE) - Windows Cloud Builds and 4 more. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 35m it lasted.

  Anything that failed between 2026-10-19 07:46 UTC and 2026-10-19 08:21 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 6

1. investigatingOct 19, 2026, 7:46 AM UTC

   We are currently investigating this issue.

   provider

2. monitoringOct 19, 2026, 8:02 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. identifiedOct 19, 2026, 8:02 AM UTC

   The issue has been identified and a fix is being implemented.

   provider

4. monitoringOct 19, 2026, 8:04 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

5. resolvedOct 19, 2026, 8:21 AM UTC

   This incident has been resolved.

   provider

6. postmortemOct 23, 2026, 6:21 PM UTC

   ## **Summary** On October 19, 2026, between approximately 12:10 AM and 12:55 AM PDT, customers on Prod4 experienced errors and intermittent unavailability of Harness Pipelines services. Heavy pipeline executions drove a sharp rise in CPU and memory use on the pipeline service, and for a short period all service instances were unavailable. Harness engineering scaled the service both vertically and horizontally, which restored service. A fix for the underlying behavior is being rolled out. ## **Root Cause** On October 19th, high number of concurrent executions with very large execution graphs drove a sharp increase in cluster resources. The service stores and retrieves this full execution graph for visualization purposes and further progressing the pipelines to next states. Our auto-scalaers on Prod4 took longer than usual for scaling resources due to a sharp rise in CPU and memory.  As a part of the fix, the execution graph update will now process graph changes incrementally instead of the full graph. It is being rolled out in a phased manner and will be completely rolled out at the earliest. ## **Impact** * Affected users: Customers on Prod4. * Symptom: unavailability of pipeline functionality in the Harness platform. * Data: No data loss. * Suggested workaround: None required from customers. Harness engineering resolved the issue. * Duration of impact: Approximately 20 minutes of full unavailability and degraded performance for 25 minutes. Harness continued monitoring after recovery and confirmed platform stability with customers. ## **Remediation** **Immediate** Harness engineering scaled the pipeline service both vertically \(more resources per instance\) and horizontally \(more instances\), and removed instances in an unhealthy state. The instance count stabilized and normal service was restored. The system will stay scaled up till the fix is completely rolled out.  **Permanent fix and Action Items** To prevent such issues from happening again, Harness is 1. Tuning the auto heal/scaling process so that impact is mimimal to none. 2. Optimizing the execution graph flow so that large graphs can be process faster.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.