Intermittent failures in Policy step evaluations on Prod 7

What to do

• Check whether your product depends on Harness. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2026-10-16 10:41 UTC and 2026-10-06 20:00 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 2

1. resolvedOct 6, 2026, 8:00 PM UTC

   On October 7, 2026, from 08:00 UTC to 11:00 UTC (1:30 PM to 4:30 PM IST), some pipeline executions on the Prod 7 cluster saw intermittent failures in Policy (OPA) step evaluations.

   provider

2. postmortemNov 10, 2026, 10:17 PM UTC

   Summary Policy evaluation for a pipeline step intermittently failed for one customer with an "unsupported principal type for internal token" error once an enhanced tenant-isolation safeguard was turned on. Root cause Our internal services authenticate to each other using signed internal tokens. The service that evaluates policy only accepts tokens whose principal is marked as one of a small set of recognized types; a token with a missing or unrecognized principal type is rejected. The affected pipeline step was missing a safeguard that other callers rely on to guarantee their internal token always carries a recognized principal type. As a result, under certain conditions it could present a token with an unsupported principal type to the policy service, causing the request to be rejected. ‌ Mitigation We mitigated the immediate impact by disabling the safeguard. Remediation We are applying the fix at the shared client layer used by every caller of the policy evaluation service, including the affected pipeline step, OPA helper utilities, secrets, and connectors, so the fix protects all current and future callers consistently. * Tokens with a recognized principal type continue to be forwarded as-is. * Tokens with a missing or unsupported principal type will instead be re-signed as the calling service. Next steps We are rolling out the fix gradually, starting in shadow mode, with additional logging and observability added across all call paths so we can confirm the fix covers every use case, including lower-traffic ones, before it is enabled broadly.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.