Intermittent External Network Connectivity Issues Affecting Build VMs

What to do

• Check whether you use Harness Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds and 1 more in us, us-central1, us-west1. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 21h 24m it lasted.

  Anything that failed between 2026-07-30 05:59 UTC and 2026-07-31 03:22 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 5

1. investigatingJul 30, 2026, 5:59 AM UTC

   Summary - We are intermittently facing network connectivity issues with our Build VM's unable to connect to external resources. We are currently investigating the issue.

   provider

2. monitoringJul 30, 2026, 8:07 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. monitoringJul 30, 2026, 8:07 AM UTC

   We are continuing to monitor for any further issues.

   provider

4. resolvedJul 31, 2026, 3:22 AM UTC

   This incident has been resolved.

   provider

5. postmortemAug 7, 2026, 7:32 PM UTC

   ## Summary CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways. ## Impact * CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services \(e.g., GitHub, Bitbucket\) over our outbound network gateways. * The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume. * No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue. * us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue. ## Root Cause ‌ Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details \(source/destination address and port\). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways ‌ ## Action Items To prevent such issues from happening again Harness will, Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.