Intermittent CI Pipeline using VM Runner slowness and errors

What to do

• Check whether your product depends on Harness. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2026-10-28 17:10 UTC and 2026-10-28 17:10 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 2

1. resolvedOct 28, 2026, 5:10 PM UTC

   Starting 10/28/2026 10:10 AM CI jobs on internal VM runners intermittently hung or expired without logs, affecting some customers in prod1/2. The resolution is to upgrade the new runner version https://github.com/drone-runners/drone-runner-aws/releases/tag/v1.0.0-rc.318 (Docker image: drone/drone-runner-aws:1.0.0-rc.318)

   provider

2. postmortemNov 10, 2026, 10:59 PM UTC

   1. Summary From October 28, 2026, 10:14 AM PT to October 29, 2026, 9:12 PM PT, CI pipelines using VM-based runners intermittently hung or expired after initialization, often with no step logs or console output. The issue affected two Prod1 customers, with additional VM-runner customers identified for proactive follow-up across Prod1, Prod2, and Prod3. The trigger was message-queue backpressure caused by broker slowness following an infrastructure image upgrade in that layer; we mitigated by rolling back that change, which resolved the platform-side trigger within about two hours. Follow-up found the two affected customers were running older versions of our VM runner and its companion log-streaming component that predated a fix for a related deadlock issue, so the customer-facing resolution was to pin to a specific, validated runner version instead of relying on a floating "latest" tag. 2. Impact * CI jobs on VM runners could hang mid-run, time out after long waits , or expire without usable logs. * This blocked deployments for affected pipelines and slowed diagnosis, since failures were often silent * Two Prod1 customers were confirmed affected; a broader set of VM-runner customers across Prod1, Prod2, and Prod3 were identified as potentially exposed and proactively contacted. * Impact was limited to CI workflows using VM-based runners; other execution types were not affected. 3. Root Cause Our CI VM-runner architecture streams step logs through a live-logging pipeline backed by a message-queue layer. During this incident, that queue experienced backpressure, following an infrastructure image upgrade on one of the queue's brokers. This degraded the live-log streaming path platform-wide, causing some in-flight log writes to stall. ‌ However, this platform-level degradation was transient and self-healed once we rolled back the image upgrade. It only turned into multi-hour, customer-visible hangs for customers who were running older builds of our VM runner and its companion log-streaming helper component,specifically, builds that predated a fix for a known deadlock condition in how that component handled interrupted log streams. Under normal queue conditions, this defect was dormant; the queue backpressure was enough to trigger it on those older builds, causing their executions to hang on an internal request until a roughly 4-hour timeout expired. Both confirmed affected customers were using runner versions older than the version containing the fix, and in one case the customer believed they were on the latest available image but were, in practice, still running an older pulled version — because our "latest" tag at the time did not reliably correspond to the newest, QA-validated build. 4. Resolution * Identified that failures were specific to VM-runner executions hanging after initialization, and separated this from an unrelated initialization-stage incident from the prior day. * Traced message-queue producer errors and broker slowness to a recent infrastructure image upgrade on that queue layer. * Rolled back that image upgrade, which brought queue/log-streaming error rates to zero within about two hours of incident start. * Confirmed, with the two affected customers, the specific runner and log-streaming component versions they were running. * Recommended and confirmed a specific, pinned runner version containing the deadlock fix as the customer-facing resolution. * Identified additional VM-runner customers for proactive outreach and validation across Prod1, Prod2, and Prod3. 5. Preventive Actions 1. Surface the VM runner version directly in pipeline execution logs and metadata, so this no longer has to be manually confirmed with customers during an incident. 2. Publish guidance recommending customers pin to an explicit runner version rather than relying on "latest". 3. Add detection for post-initialization stalls, specifically keyed to the ~4-hour timeout signature seen in this incident. 4. Build a queryable inventory of VM-runner customers and the versions they're running, to support faster, more targeted response in future incidents and proactive upgrade outreach.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.