Pipelines are stuck in Prod1

What to do

• Check whether you use Harness Continuous Delivery - Next Generation (CDNG). Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 40m it lasted.

  Anything that failed between 2026-09-03 10:00 UTC and 2026-09-03 11:40 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 4

1. investigatingSep 3, 2026, 10:00 AM UTC

   Pipeline execution is stuck in Prod1. We are investigating the issue.

   provider

2. monitoringSep 3, 2026, 10:28 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. resolvedSep 3, 2026, 11:40 AM UTC

   This incident has been resolved.

   provider

4. postmortemSep 10, 2026, 7:21 AM UTC

   # Summary On September 3, 2026, between approximately 2:54 AM and 3:44 AM PDT, customers running pipelines on Prod1 experienced delays in pipeline execution graph rendering and execution status updates. Pipeline execution itself was not affected — pipelines continued to run and complete — but the visual graph and status information in the Harness UI lagged behind actual execution progress. Harness engineering identified the cause, added processing capacity, and restored normal graph and status updates. Extended monitoring confirmed full recovery the following day. # Root Cause A planned database failover activity in the Prod1 region temporarily increased network latency between the pipeline execution service and its database while traffic was briefly served cross-region. This reduced the rate at which the service could process pipeline orchestration events, and a processing backlog began to build. Pipeline execution graphs and status indicators in the UI depend on these orchestration events being processed in near real time. As the backlog grew, graph rendering and status updates fell increasingly behind actual execution progress, and some graphs had to be rebuilt rather than served from cache. Pipeline execution itself does not depend on this same processing path, so pipelines continued to run and complete throughout the incident. # Impact * Affected users: Customers with pipeline executions running on Prod1 during the incident window. * Symptom: Delayed pipeline execution graph rendering and delayed execution status updates in the Harness UI. * Not reported as impacted: Pipeline execution itself — pipelines continued to run and complete. * Data: No data loss, corruption, or exposure was identified. * Suggested workaround: None required from customers. The issue was resolved entirely by Harness engineering. * Duration of impact: Approximately 45–50 minutes of degraded graph and status visibility, from 2:54 AM to 3:44 AM PDT on September 3, 2026. Harness continued monitoring after the backlog cleared and confirmed full recovery. # Remediation ## Immediate Harness engineering added processing capacity to the affected service and restarted it, which cleared the event-processing backlog and restored normal pipeline execution graph and status updates. # Action Items | **Action Item** | **Objective** | | --- | --- | | Queue based alarms | We are raising priority for queue based alarms so we can take faster remediation actions \(increase capacity\) |

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.