We are currently seeing delegate selection issue for VM infra executions

What to do

• Check whether you use Harness Continuous Integration Enterprise(CIE) - Self Hosted Runners. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 32m it lasted.

  Anything that failed between 2026-10-27 19:52 UTC and 2026-10-27 22:24 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 6

1. investigatingOct 27, 2026, 7:52 PM UTC

   We are currently investigating this issue.

   provider

2. investigatingOct 27, 2026, 7:58 PM UTC

   CI pipelines on VM infrastructure are failing at the Initialize step because the task goes to a delegate outside the intended pool

   provider

3. identifiedOct 27, 2026, 9:15 PM UTC

   The issue has been identified and a fix is being implemented.

   provider

4. monitoringOct 27, 2026, 10:16 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

5. resolvedOct 27, 2026, 10:24 PM UTC

   This incident has been resolved.

   provider

6. postmortemNov 10, 2026, 10:55 PM UTC

   1. Summary On October 27, 2026, starting around 6:35 AM, customers running CI pipelines on self-managed VM infrastructure \(a legacy Delegate-based VM runner configuration\) in Prod1 began experiencing pipeline failures caused by tasks being routed to Delegates that could not actually execute them. The issue affected a small number of customers using this specific, less common configuration \(VM-based CI without Delegate tags configured\). It was introduced by a backend deployment and was mitigated by rolling that deployment back; a permanent fix followed. 2. Impact * CI pipelines using VM-based execution without Delegate tags configured began intermittently failing to run on affected Delegates. * Other pipeline types and configurations were not affected. * Impact was limited to a small number of customers in Prod1 using this specific configuration. * The partial outage lasted approximately 8 hours before being resolved via rollback. Affected customers were given a configuration-based workaround \(setting Delegate tags\) during the incident. 3. Root Cause Our Delegate platform uses a feature called Capability Checks to determine whether a given Delegate is actually able to execute a specific task, for example, whether it has access to the right infrastructure or connector. When Delegate tags are explicitly configured, this check matters less, but for configurations without tags, it is the primary mechanism for routing work correctly. ‌ As part of ongoing work to support capability checking for a newer Delegate version, an optimization was introduced that skipped running a capability check whenever its underlying validation criteria came back empty, the assumption being that an empty criteria value meant there was nothing meaningful to validate, so the Delegate could be treated as already capable. ‌ In practice, several existing capability checks had previously harmless defect, which under certain conditions would produce an empty criteria value despite still having a meaningful check that should run. Skipping checks in that state caused Delegates to be automatically treated as capable regardless of whether they actually were and most significantly, for legacy CI VM pipelines, where this logic determines whether a given Delegate is managing the correct virtual machine pool. With the check skipped, Delegates were marked capable of managing any VM pool, which caused CI tasks to be routed to Delegates that could not actually run them. ‌ Because the underlying defects were pre-existing but previously harmless, the issue only surfaced once the new optimization began skipping the checks that had been quietly masking them, and it took several hours to appear widely because of a caching layer with a 6-hour refresh interval. 4. Resolution * Identified that CI VM task routing was being driven by faulty capability-check results. * Rolled back the backend deployment containing the optimization \(the first rollback attempt did not fully take effect; a retry succeeded\). * Provided customers with a configuration-based workaround \(explicitly setting Delegate tags\) to reduce impact while the rollback was in progress. * Reverted the capability-check optimization so checks with empty criteria run normally again rather than being skipped. * Added metrics and logging to help identify the specific capability checks that were producing empty criteria incorrectly, so they can be fixed at the source. 5. Preventive Actions 1. Fix the underlying capability checks that were producing empty criteria incorrectly \(several affecting CI VM pipelines, with a few additional minor cases identified\). 2. Safely re-introduce the capability-check optimization once those checks are fixed, since empty criteria should not normally occur once the underlying defects are resolved. 3. Add automated test coverage that exercises this CI VM Delegate-routing use case, since this was a regression that automated tests did not catch. 4. Add alerting specific to CI VM infrastructure routing, since no automated alert fired for this incident and it was first identified by customers.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.