Prod2 was intermittently unavailable

What to do

• Check whether you use Harness Platform. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1m it lasted.

  Anything that failed between 2026-09-05 09:40 UTC and 2026-09-05 09:40 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 3

1. investigatingSep 5, 2026, 9:40 AM UTC

   We are currently investigating this issue.

   provider

2. resolvedSep 5, 2026, 9:40 AM UTC

   This incident has been resolved.

   provider

3. postmortemSep 9, 2026, 6:22 PM UTC

   ## **Summary** Between 12:34am PST and 12:38am PST on 5th September, the Delegate service manager experienced some elevated exceptions when attempting to write to the database. Consequently, delegate connections were dropped, causing them to disconnect. Delegate automatically re-attempts registration back to the `delegate service manager`  and majority of the delegates got connected back after the incident. For Docker and ECS delegates the automatic restart is not enabled unless these delegates have health monitoring enabled. For these delegates a manual restart is needed and was recommended.  Post restart the delegate would re-connect and the issue was resolved. ## **Root cause** On Prod2 cluster we identified a performance bottleneck in the delegate service that, under certain conditions, can increase database write latency and delay heartbeat processing which leads to delegates being disconnected. ## **Impact** All K8s delegates and \`Docker/ECS\` delegates got connected back immediately within 4 mins and started to function normally. The impact can be scoped to those specific types of delegates that didn’t have health monitoring enabled.  ## **Remediation** * Immediate: We have added additional monitoring and increased resources for handling the influx of traffic.  * Permanent: We have identified a hotspot in the code that can cause high latency when writing to a database which we are actively working on resolving.  ## **Action Items** To prevent such issues from happening again, Harness will work on the following: 1. Increased targeted monitoring and alerting to initiate timely mitigation and prevent this from happening again. 2. Fix the identified delegate service managers database client reconnect failures 3. Fix the hotpots that can cause query latency.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.