Prod2 Pipelines with verify step failing intermittently

What to do

• Check whether you use Harness Continuous Delivery - Next Generation (CDNG). Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 41m it lasted.

  Anything that failed between 2026-10-06 12:54 UTC and 2026-10-06 15:35 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 3

1. monitoringOct 6, 2026, 12:54 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

2. resolvedOct 6, 2026, 3:35 PM UTC

   This incident has been resolved.

   provider

3. postmortemOct 10, 2026, 1:57 AM UTC

   ### Summary Customers on Prod 2 experienced intermittent failures of a few Verify steps at startup. Affected steps failed shortly after launch with a message indicating Continuous Verification\(CV\) could not reach an internal Harness service. The issue stemmed from a recent service update that introduced connection pooling to improve performance and reliability with a keep-alive setting incompatible with the receiving service's timeout. No customer data was lost or corrupted. ### Customer Impact Customers using CV Verify steps on Prod-2 saw intermittent step failures during the incident window. The failure rate was low — the majority of executions completed successfully throughout the window. Pipelines that did not use CV Verify steps were unaffected. ### Root Cause A recent CV service update introduced connection pooling to an internal platform component, with a keep-alive duration longer than that component's idle timeout. Connections idle long enough were silently closed by the server; the CV service did not detect this and had no retry logic for such failures, causing them to surface directly as Verify step failures. The issue was intermittent because it only affected connections that happened to be idle long enough to expire. ### Mitigation The connection keep-alive duration was reduced to fall within the receiving service's idle timeout. Failures dropped to near zero immediately, and no further step failures of this type have been observed on any production environment. ### Next Steps To prevent recurrence, Harness will: * **Fix connection handling:** Tuned the default keep-alive and enabled automatic retry on connection failures so stale-connection errors are handled transparently. * **Add resilience at step startup:** Introduce retry with backoff for the internal service call at the Verify step start, so isolated transient failures do not surface as immediate step failures. * **Improve alerting:** Enhanced monitoring for internal connection error rates.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.