Pipelines are failing for harness IACM customers

What to do

• Check whether you use Harness Infrastructure as Code Management (IaCM), Infrastructure as Code Management (IaCM), Infrastructure as Code Management (IaCM), Infrastructure as Code Management (IaCM). Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 32m it lasted.

  Anything that failed between 2026-08-26 08:38 UTC and 2026-08-26 09:10 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 5

1. investigatingAug 26, 2026, 8:38 AM UTC

   We are currently investigating an issue reported in IACM pipelines in Prod-1 , Prod-2 ,Prod-4 and EU1 harness clusters.

   provider

2. investigatingAug 26, 2026, 8:43 AM UTC

   We are continuing to investigate this issue.

   provider

3. monitoringAug 26, 2026, 9:05 AM UTC

   We have reverted the change that caused this issue in all clusters.

   provider

4. resolvedAug 26, 2026, 9:10 AM UTC

   This incident has been resolved.

   provider

5. postmortemSep 21, 2026, 6:07 PM UTC

   # Root Cause Analysis Report # 1. Summary On Wednesday, August 26, 2026, some customers running Infrastructure as Code Management \(IaCM\) pipelines in Harness production saw those pipelines start, sit in a Running state with no init or plan steps, then abort or expire after about 10 minutes. OpenTofu / Terraform work did not start. The failure presented as a pipeline timeout; it was not an OpenTofu timeout and not a failure of the customer's infrastructure-as-code. Other product areas that do not go through this IaCM-to-Pipeline handoff were not affected by this failure mode. # 2. Impact * Some IaCM pipelines did not complete. Stages appeared stuck, then timed out or aborted. * Execution graphs showed the IaCM stage Running with no child steps. Init and plan never ran, so there was no plan output and no apply. * A reported affected execution ran for 10 minutes and 7 seconds before being aborted. That interval is a platform stuck-execution safeguard, not a customer-configured step-timeout setting. * Customers needed to re-run their pipeline after service was restored in their environment. # 3. Root Cause IaCM stages are driven by the Harness Pipeline service. After an IaCM stage starts, IaCM must tell Pipeline to create the child steps \(init, plan, and so on\). Those messages travel on Harness's internal event bus. An incompatible modification within a shared library \(unrelated to IaCM specifically\) was inherited by the IaCM Manager upon rebuilding with the updated version, without any direct changes to IaCM functionality. # 5. Remediation IaCM Manager was rolled back to an older version in each affected production environment, All affected environments were restored by 09:00:40 UTC on August 26. Customers do not need to change pipeline YAML, OpenTofu/Terraform configuration, or connectors. Re-running the same pipeline after service was restored is sufficient. # 6. Preventive Actions 1. **Backward-compatible or atomic SDK/event changes:** producer and consumer changes in the shared Pipeline SDK and Pipeline service must remain backward compatible, or be shipped and deployed together atomically, so that mixed versions can never silently drop messages. 2. **Detection:** add alerting for an IaCM stage that stays Running with zero children, or for child-step events that are produced but never consumed, rather than relying on the time based abort as the first signal.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.