Degraded CI performance

What to do

• Check whether you use Harness Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Integration Enterprise(CIE) - Windows Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 4h 40m it lasted.

  Anything that failed between 2026-07-17 17:16 UTC and 2026-07-17 21:56 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 5

1. investigatingJul 17, 2026, 5:16 PM UTC

   We are currently investigating this issue.

   provider

2. identifiedJul 17, 2026, 5:34 PM UTC

   The issue has been identified and a fix is being implemented.

   provider

3. monitoringJul 17, 2026, 6:12 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

4. resolvedJul 17, 2026, 9:56 PM UTC

   This incident has been resolved.

   provider

5. postmortemAug 4, 2026, 2:48 AM UTC

   # Summary On 17 July 2026, following a routine code deployment, customers on older delegate versions \(858xx and below\) began experiencing delayed CI builds on Harness Cloud-hosted builds using our global build-queueing capability. Affected builds experienced an unexpected pause of up to approximately 8 minutes at the "waiting for infrastructure" stage before continuing, rather than proceeding within the expected sub-second time. Overall build slowness was intermittent. # Impact * All CI builds were potentially subject to delay; impact was most pronounced for builds on Harness Cloud-hosted infrastructure using the global build-queueing feature. * Affected builds experienced an unexplained pause of up to approximately 8 minutes before continuing, followed by a slower "cold start" since a pre-reserved compute slot was not available — this presented to users as slow builds rather than build failures. * Accounts running on newer delegate versions \(858xx and above\) were not impacted. * No builds failed outright as a direct result of this issue, and no data was lost. # Root Cause The root cause was an internal code change that inadvertently broke how a specific build-queueing record was read back from our database once builds that had already been queued under the previous version of the code encountered the newly deployed version. We resolved the immediate impact by cleaning up the affected records and reverting the underlying code change, and we are implementing several safeguards to prevent this class of issue from recurring. ‌ # Next Steps We assess the risk of a similar recurrence as low as the following actions are being understaken.The specific code path that caused this incident has already been reverted, and we are implementing structural safeguards so that this general class of issue cannot recur, regardless of where in the codebase it might otherwise occur. | **Corrective / Preventive Action** | | --- | | Add explicit, stable identifiers to all internal data classes that get stored in our database, so that future internal code reorganizations cannot break the system's ability to read back previously stored records. | | Introduce rollback and backward-compatibility testing in our pre-production environment, specifically designed to catch this class of issue before it reaches production. |

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.