Single Customer: IaCM pipeline executions failing intermittently in Prod 1

What to do

• Check whether you use Harness Infrastructure as Code Management (IaCM). Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 4h 22m it lasted.

  Anything that failed between 2026-10-06 11:58 UTC and 2026-10-06 16:21 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 5

1. investigatingOct 6, 2026, 11:58 AM UTC

   We are currently investigating this issue.

   provider

2. investigatingOct 6, 2026, 11:59 AM UTC

   We are continuing to investigate this issue.

   provider

3. identifiedOct 6, 2026, 12:24 PM UTC

   The issue has been identified and a fix is being implemented.

   provider

4. resolvedOct 6, 2026, 4:21 PM UTC

   This incident has been resolved.

   provider

5. postmortemOct 9, 2026, 6:48 PM UTC

   ## Summary On **Tuesday, 6 October 2026**, IaCM pipelines that use the **OpenTofu plugin** in **Prod1** failed intermittently for a production accountAffected runs injected secrets successfully, then later steps timed out or failed authentication. The same pipeline often succeeded on retry, with no configuration change. Typical failing executions ran for about 90 minutes. Other Harness modules were not broadly impacted. We have no report of customer data loss. Pipelines that finished inside the token lifetime were unaffected by this defect. ## Root Cause The IaCM stages minted authentication tokens that expired after one hour. Policy checks later in a long run then rejected the token, so the step timed out or failed authentication. That one-hour lifetime was fixed. Runs of about 90 minutes crossed the one-hour mark, which is why the failures showed up on longer pipelines and often cleared on a shorter retry. A separate set of plan errors in the same account came from credential setup in the customer workspace and is outside this incident. ## Impact * **Product area:** IaCM pipelines using the IaCM plugins talking to OPA sdk. * **Customer experience:** intermittent timeouts and authentication failures after secret injection, on runs longer than about one hour. * **Environment:** Prod1, one production account * **Data:** No data loss reported. ‌ ## Remediation * **Immediate:** Opened a status page for the affected Prod1 account, and shipped an interim IaCM manager change for one class of the observed failures while the plugin fix was built. * **Permanent:** Shipped an OpenTofu plugin hotfix so token expiry is configurable. In production the lifetime was raised to about 6 hours, so a long pipeline does not outlive the token. The hotfix was validated in QA and deployed to Prod1. We apologise for the disruption caused. Please reach out to your Harness support contact if you have any further questions.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.