DBDevOps Builds Failing intermittently

What to do

• Check whether you use Harness DBDevOps, DBDevOps. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1d 7h it lasted.

  Anything that failed between 2026-09-17 14:00 UTC and 2026-09-18 21:30 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 7

1. investigatingSep 17, 2026, 2:00 PM UTC

   We are currently investigating this issue.

   provider

2. investigatingSep 17, 2026, 4:58 PM UTC

   We are continuing to investigate this issue.

   provider

3. identifiedSep 17, 2026, 5:13 PM UTC

   The issue has been identified and a fix is being implemented.

   provider

4. monitoringSep 17, 2026, 8:05 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

5. monitoringSep 17, 2026, 8:11 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

6. resolvedSep 18, 2026, 9:30 PM UTC

   This incident has been resolved.

   provider

7. postmortemSep 22, 2026, 7:32 PM UTC

   ## Summary Between approximately **8:40 AM and 1:00 PM PDT on Wednesday, September 17, 2026**, some customers running **Database DevOps** pipelines in Production experienced intermittent failures when executing schema migration steps \(for example Apply Schema and related Database DevOps steps\). Affected runs commonly surfaced a misleading error: ‌ `Invalid request: Unable to fetch port details. Please verify pipeline yaml and check if 'Enable container based execution' toggle is on for the step group.` ‌ Failures were intermittent as the same pipeline could succeed some times. Other Harness modules and non–Database DevOps CD pipelines were not broadly impacted. ‌ ‌ ## Root Cause This issue required two separate problems to occur together. First, pipeline plan creation runs on a shared pool of background worker threads, and each thread can handle work for many different customer accounts over time. The identity used to authenticate outbound service calls was tied to the thread rather than to the specific request, so a thread could occasionally carry over the identity from a previous account's request instead of using the correct one for the current request. This happened inconsistently as the same pipeline could compile fine on one run and pick up the wrong identity on the next, which is why the issue appeared and disappeared without any customer-side change, and why retries usually worked. ‌ Second, and independently, a recent change to our Database DevOps service added a stricter check to make sure the identity on an incoming request matches the account it claims to represent. Before this change, a mismatched identity like the one above would have gone unnoticed and the request would still succeed. After this change, that same mismatch was correctly rejected, but the calling service treated that rejection as a hard failure rather than retrying, which is what turned an internal identity mix-up into a visible pipeline failure for customers. ‌ Neither problem caused issues on its own. The identity mix-up was harmless until the stricter check existed to catch it, and the stricter check only became noticeable because the identity mix-up was already happening in the background ‌ ## Impact * **Product area:** Database DevOps pipeline steps \(schema apply and related migration steps\) in Production. * **Customer experience:** Intermittent pipeline failures. Some executions succeeded while others failed within minutes of each other. The reported error pointed at container/port configuration and was misleading relative to the actual cause. Retries frequently succeeded. * **Scope:** Limited to customers running Database DevOps workloads on Production clusters where the stricter account validation was active. Other CD and platform functionality was not broadly disrupted, and no data loss or unintended schema changes resulted , affected executions failed rather than applying incorrect changes. ‌ ## Remediation * **Immediate:** Rolled back the Database DevOps service change that enforced the stricter account-binding validation in the most impacted Production environment, restoring Database DevOps pipeline execution for affected customers. * **Permanent:** Shipped a fix that explicitly sets the correct source principal and restores the previous thread context afterwards around every outbound call made by Database DevOps plan creators, including the plugin capability call. Outbound calls now always carry the account context of the pipeline being compiled, regardless of which worker thread runs it. * **Verification:** Following the rollout, service logs showed no further account-mismatch rejections across monitored clusters. ## Action Items ‌ ‌ To prevent such issues from happening again, Harness will,  * Enhance tests that compile plans for two accounts on a shared executor and assert each outbound call carries the correct account. * **Fail-fast on capability-check auth errors**: treat 403/auth failures as hard errors with an accurate message, instead of silently falling back to the legacy execution path. * **Alerting on account-binding rejections and plugin-capability 403 rate**: surface these signals before customer impact

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.