# Prod1: IACM Stages are failing and pipelines are  intermittently failing to start.

- Provider: [Harness](https://down.guide/harness)
- Level: confirmed
- Impact: Partial outage
- lasted 58m
- started 2026-10-06 21:46 UTC
- Infrastructure as Code Management (IaCM)
- id: DWN-43ce337195
- Official page: https://stspg.io/97z6j192pz3j

## Root cause
> The issue was triggered by an unusually large burst of log activity from one customer's pipeline.

[Postmortem](https://stspg.io/97z6j192pz3j)

## Updates from Harness
### 2026-10-06 21:46 UTC: investigating
We are currently investigating this issue.

### 2026-10-06 21:47 UTC: investigating
We are continuing to investigate this issue.

### 2026-10-06 22:03 UTC: identified
The issue has been identified and a fix is being implemented.

### 2026-10-06 22:30 UTC: monitoring
A fix has been implemented and we are monitoring the results.

### 2026-10-06 22:30 UTC: monitoring
A fix has been implemented and we are monitoring the results. we have confirmation of this fixing the issue.

### 2026-10-06 22:44 UTC: resolved
This incident has been resolved.

### 2026-10-23 02:12 UTC: postmortem
# 1. Summary
On October 6, 2026, pipelines containing an IaCM stage were unable to start on Prod1 for approximately two hours. Affected users saw an error when the pipeline attempted to create its execution plan. Pipelines that did not use an IaCM stage continued to run normally throughout.
The underlying cause was an internal logging defect in the service that handles IaCM stage execution: under a specific, rare timing condition, that service's own logging system could end up waiting on itself indefinitely, which in turn stalled the broader process responsible for preparing pipeline execution plans. The issue was triggered by an unusually large burst of log activity from one customer's pipeline. We resolved the immediate impact by rolling the service back to its previous stable version, and a permanent code fix is currently in testing.
# 2. Impact
\* All pipelines using an IaCM stage on Prod1 were unable to start for approximately two hours.
\* Multiple customers on the Prod1 cluster were affected.
\* Pipelines that did not use an IaCM stage were not affected and continued running normally.
\* No data was lost as a result of this incident.
\* As a side effect of the rollback used to resolve this incident, an unrelated fix for one customer \\(extending how long a certain authentication token remains valid during long-running pipelines\\) was temporarily reverted. That fix was reapplied the same evening.
# 3. Root Cause
The service responsible for IaCM pipeline stages logs its activity through an internal queue: log messages are placed on the queue, and a single background worker continuously reads from it and writes the messages out. Separately, that service has a filter that can rewrite the severity of certain log messages back onto that same queue.
Under a specific, narrow timing condition, this created a deadlock, if the queue happened to be completely full at the exact moment a rewritten message was being submitted back onto it, the one worker responsible for freeing up space on the queue would itself get stuck waiting for space to free up, a condition it could never resolve, since it was the only thing that could have freed that space. Once that happened, every other part of the service that needed to log anything also became stuck waiting, which cascaded into the pipeline-plan-creation process failing for any pipeline using an IaCM stage.
# 4. Resolution
\* Rolled the service back to its last stable version, which resolved the immediate incident.
# 5. Preventive Actions
1. \*\*Optimize the underlying defect\*\*: Optimize and tune the logging filter so that rewriting a message's severity happens on the thread that generated the log line, rather than being resubmitted through the same background worker's queue, eliminating the deadlock condition entirely.
2. \*\*Enhance monitoring on logging-queue health\*\*: add metrics and alerting on this internal queue so that this class of saturation pages our on-call team automatically rather than relying on customer reports.
3. \*\*Evaluate limits and perform high stress tests with very large pipelines\*\*:

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).