# We are currently seeing delegate selection issue for VM infra executions - Provider: [Harness](https://down.guide/providers/harness) - Confidence: Confirmed - Impact: Degraded - lasted 2h 32m - started 2026-10-27 19:52 UTC - Continuous Integration Enterprise(CIE) - Self Hosted Runners - id: DWN-3dc6714aeb - Official page: https://stspg.io/xsr0cq7g8zjm ## Updates from Harness ### 2026-10-27 19:52 UTC: investigating We are currently investigating this issue. ### 2026-10-27 19:58 UTC: investigating CI pipelines on VM infrastructure are failing at the Initialize step because the task goes to a delegate outside the intended pool ### 2026-10-27 21:15 UTC: identified The issue has been identified and a fix is being implemented. ### 2026-10-27 22:16 UTC: monitoring A fix has been implemented and we are monitoring the results. ### 2026-10-27 22:24 UTC: resolved This incident has been resolved. ### 2026-11-10 22:55 UTC: postmortem 1. Summary On October 27, 2026, starting around 6:35 AM, customers running CI pipelines on self-managed VM infrastructure \\(a legacy Delegate-based VM runner configuration\\) in Prod1 began experiencing pipeline failures caused by tasks being routed to Delegates that could not actually execute them. The issue affected a small number of customers using this specific, less common configuration \\(VM-based CI without Delegate tags configured\\). It was introduced by a backend deployment and was mitigated by rolling that deployment back; a permanent fix followed. 2. Impact \* CI pipelines using VM-based execution without Delegate tags configured began intermittently failing to run on affected Delegates. \* Other pipeline types and configurations were not affected. \* Impact was limited to a small number of customers in Prod1 using this specific configuration. \* The partial outage lasted approximately 8 hours before being resolved via rollback. Affected customers were given a configuration-based workaround \\(setting Delegate tags\\) during the incident. 3. Root Cause Our Delegate platform uses a feature called Capability Checks to determine whether a given Delegate is actually able to execute a specific task, for example, whether it has access to the right infrastructure or connector. When Delegate tags are explicitly configured, this check matters less, but for configurations without tags, it is the primary mechanism for routing work correctly. ‌ As part of ongoing work to support capability checking for a newer Delegate version, an optimization was introduced that skipped running a capability check whenever its underlying validation criteria came back empty, the assumption being that an empty criteria value meant there was nothing meaningful to validate, so the Delegate could be treated as already capable. ‌ In practice, several existing capability checks had previously harmless defect, which under certain conditions would produce an empty criteria value despite still having a meaningful check that should run. Skipping checks in that state caused Delegates to be automatically treated as capable regardless of whether they actually were and most significantly, for legacy CI VM pipelines, where this logic determines whether a given Delegate is managing the correct virtual machine pool. With the check skipped, Delegates were marked capable of managing any VM pool, which caused CI tasks to be routed to Delegates that could not actually run them. ‌ Because the underlying defects were pre-existing but previously harmless, the issue only surfaced once the new optimization began skipping the checks that had been quietly masking them, and it took several hours to appear widely because of a caching layer with a 6-hour refresh interval. 4. Resolution \* Identified that CI VM task routing was being driven by faulty capability-check results. \* Rolled back the backend deployment containing the optimization \\(the first rollback attempt did not fully take effect; a retry succeeded\\). \* Provided customers with a configuration-based workaround \\(explicitly setting Delegate tags\\) to reduce impact while the rollback was in progress. \* Reverted the capability-check optimization so checks with empty criteria run normally again rather than being skipped. \* Added metrics and logging to help identify the specific capability checks that were producing empty criteria incorrectly, so they can be fixed at the source. 5. Preventive Actions 1. Fix the underlying capability checks that were producing empty criteria incorrectly \\(several affecting CI VM pipelines, with a few additional minor cases identified\\). 2. Safely re-introduce the capability-check optimization once those checks are fixed, since empty criteria should not normally occur once the underlying defects are resolved. 3. Add automated test coverage that exercises this CI VM Delegate-routing use case, since this was a regression that automated tests did not catch. 4. Add alerting specific to CI VM infrastructure routing, since no automated alert fired for this incident and it was first identified by customers. From the builder: does your app depend on Harness? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=harness) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Harness. --- down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-3dc6714aeb), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.