Degraded performance in UI execution graph in prod3

What to do

• Check whether you use Harness Continuous Delivery - Next Generation (CDNG). Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 11m it lasted.

  Anything that failed between 2026-10-15 08:04 UTC and 2026-10-15 08:15 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 5

1. investigatingOct 15, 2026, 8:04 AM UTC

   We are currently investigating this issue.

   provider

2. identifiedOct 15, 2026, 8:05 AM UTC

   The issue has been identified and a fix is being implemented.

   provider

3. monitoringOct 15, 2026, 8:09 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

4. resolvedOct 15, 2026, 8:15 AM UTC

   This incident has been resolved.

   provider

5. postmortemNov 10, 2026, 10:24 PM UTC

   Root Cause Analysis Report 1. Summary On October 15, 2026, between 12:24 AM and 1:16 AM PT, Prod3 experienced a delay in how quickly execution graphs, aggregations, and observability data updated. The underlying workflow execution engine continued to function normally throughout and note that, this was a data-freshness and UI-responsiveness issue, not a workflow execution failure. The root cause was a backend search-service dependency in our event-processing pipeline that became a bottleneck under an unusually high volume of API calls to a specific execution-details endpoint, concentrated from one customer account. We mitigated by scaling the search cluster, and after a brief recurrence later that morning, doubled its capacity. We subsequently shipped a permanent fix that removed the search-service dependency from this code path entirely, cutting search load by about 95%. 2. Impact * Prod3 users may have seen delayed or stale execution graphs, aggregation views, and observability data for about 52 minutes, with a brief recurrence later the same morning. * Underlying workflow execution was not affected and pipelines continued to run and complete normally throughout. * No confirmed workflow execution failures occurred, and no data was lost. * We temporarily applied protective rate-limiting safeguards on the affected API and coordinated with the customer whose traffic drove the load while a permanent fix was developed and deployed. 3. Root Cause Our platform processes workflow execution events through an internal, queue-based pipeline. A step in that pipeline performed a synchronous search-service lookup as part of completing each event which is aa dependency that, under normal conditions, added negligible delay. ‌ An unusually high and sustained volume of calls to an execution-details API, concentrated from one customer account, drove a large increase in search-service query volume. This pushed CPU on the search cluster's warm-tier nodes to saturation, which in turn slowed the synchronous lookups in the event-processing pipeline, causing events to back up and execution-graph and observability data to fall behind in near-real time, even though underlying workflow execution did not depend on this same path. 4. Resolution * Verified core databases and recent deployments were not responsible, narrowing the investigation to the search-service dependency. * Identified and mitigated warm-tier search-cluster CPU saturation by scaling cluster capacity during the incident. * Further increased search-cluster capacity after a brief recurrence later the same morning. * Added temporary logging to better identify which traffic was driving load, and worked with the customer to understand their usage pattern. * Applied temporary protective rate-limiting on the affected endpoint as a safeguard. * Shipped a permanent API optimization that removes the search-service dependency from this workflow-event-processing path, which reduced search load by about 95%. 5. Preventive Actions 1. Add duration-based alerting for sustained search-cluster CPU saturation, correlated with event-processing lag, so prolonged stress escalates automatically rather than depending on manual observation. 2. Validate search-cluster capacity headroom going forward, and evaluate whether capacity can now be safely reduced given the dependency has been removed from the hot path. 3. Decouple or bound similar dependencies: audit other workflow-event-processing paths for synchronous search-service calls and inventory similar dependencies elsewhere in the platform. 4. Add finer-grained rate limiting on the affected endpoint so that one customer's traffic pattern cannot affect other customers sharing the same infrastructure. 5. Define a clearer policy for applying protective traffic controls when a single customer's usage threatens shared-platform stability. ‌ ‌ _We appreciate your patience while we worked through a complete resolution, and we remain committed to strengthening the resilience of our platform as a result of this incident._

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.