Investigating CI Pipeline Delays

What to do

• Check whether you use Harness Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 58m it lasted.

  Anything that failed between 2026-09-22 16:40 UTC and 2026-09-22 18:38 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 5

1. investigatingSep 22, 2026, 4:40 PM UTC

   We are currently investigating this issue.

   provider

2. identifiedSep 22, 2026, 5:14 PM UTC

   The issue has been identified and a fix is being implemented.

   provider

3. monitoringSep 22, 2026, 6:01 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

4. resolvedSep 22, 2026, 6:38 PM UTC

   This incident has been resolved.

   provider

5. postmortemOct 2, 2026, 10:05 PM UTC

   # Summary On September 22, 2026, three of our outbound network gateways in one region became unhealthy and began rejecting a small percentage of new outbound connection attempts. The issue was resolved the same day problem completely with no further recurrence or follow-on impact.The underlying cause was a saturated internal traffic-monitoring buffer on those gateways, which caused their health checks to respond too slowly and get marked unhealthy.  # Impact * Only three gateway instances in one region were affected; other regions and other gateways in the same region were not impacted. * There was no cascading failure , the issue was fully resolved with a single round of gateway restarts, and no follow-on issues occurred afterward. # Root Cause  Our outbound network gateways run an internal component that monitors network traffic for operational visibility. Under sustained high traffic, this component's internal buffer became saturated, which created backpressure severe enough to slow down the gateway's own health-check responses beyond the timeout our load balancer allows. Once that timeout was exceeded, the load balancer marked the affected gateways unhealthy and began rejecting a portion of new connections destined for them. Actual packet forwarding continued to work normally throughout the incident. This was specifically a health-check and monitoring-buffer issue, not a failure of the gateways' core forwarding capability. # Resolution The three affected gateway instances were manually restarted, which cleared the saturated monitoring buffer and immediately restored normal health-check behavior. Connection rejections stopped as soon as the restart completed, and no further intervention was needed. # Preventive Actions | **Action** | | --- | | Replace the current traffic-monitoring approach on all gateways with a more efficient, lower-overhead implementation that cannot saturate in this way, and that alerts well before any customer-visible impact. | | Implement automatic, periodic replacement of long-running gateway instances on a fixed schedule, so no instance can accumulate the kind of age-related degradation seen in this incident. | | Migrate gateway instances to network-optimized infrastructure with substantially more dedicated network capacity, better suited to sustained high-throughput traffic. | | Enhance existing monitoring buffer's utilization while the longer-term replacement is rolled out. | | Enhance alerting specifically on gateway health-check response latency, to catch early signs of degradation before it becomes customer-visible. |

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.