# Degraded performance in UI execution graph in prod3 - Provider: [Harness](https://down.guide/providers/harness) - Confidence: Confirmed - Impact: Degraded - lasted 11m - started 2026-10-15 08:04 UTC - Continuous Delivery - Next Generation (CDNG) - id: DWN-7155f9d112 - Official page: https://stspg.io/8q9r632ktw3c ## Updates from Harness ### 2026-10-15 08:04 UTC: investigating We are currently investigating this issue. ### 2026-10-15 08:05 UTC: identified The issue has been identified and a fix is being implemented. ### 2026-10-15 08:09 UTC: monitoring A fix has been implemented and we are monitoring the results. ### 2026-10-15 08:15 UTC: resolved This incident has been resolved. ### 2026-11-10 22:24 UTC: postmortem Root Cause Analysis Report 1. Summary On October 15, 2026, between 12:24 AM and 1:16 AM PT, Prod3 experienced a delay in how quickly execution graphs, aggregations, and observability data updated. The underlying workflow execution engine continued to function normally throughout and note that, this was a data-freshness and UI-responsiveness issue, not a workflow execution failure. The root cause was a backend search-service dependency in our event-processing pipeline that became a bottleneck under an unusually high volume of API calls to a specific execution-details endpoint, concentrated from one customer account. We mitigated by scaling the search cluster, and after a brief recurrence later that morning, doubled its capacity. We subsequently shipped a permanent fix that removed the search-service dependency from this code path entirely, cutting search load by about 95%. 2. Impact \* Prod3 users may have seen delayed or stale execution graphs, aggregation views, and observability data for about 52 minutes, with a brief recurrence later the same morning. \* Underlying workflow execution was not affected and pipelines continued to run and complete normally throughout. \* No confirmed workflow execution failures occurred, and no data was lost. \* We temporarily applied protective rate-limiting safeguards on the affected API and coordinated with the customer whose traffic drove the load while a permanent fix was developed and deployed. 3. Root Cause Our platform processes workflow execution events through an internal, queue-based pipeline. A step in that pipeline performed a synchronous search-service lookup as part of completing each event which is aa dependency that, under normal conditions, added negligible delay. ‌ An unusually high and sustained volume of calls to an execution-details API, concentrated from one customer account, drove a large increase in search-service query volume. This pushed CPU on the search cluster's warm-tier nodes to saturation, which in turn slowed the synchronous lookups in the event-processing pipeline, causing events to back up and execution-graph and observability data to fall behind in near-real time, even though underlying workflow execution did not depend on this same path. 4. Resolution \* Verified core databases and recent deployments were not responsible, narrowing the investigation to the search-service dependency. \* Identified and mitigated warm-tier search-cluster CPU saturation by scaling cluster capacity during the incident. \* Further increased search-cluster capacity after a brief recurrence later the same morning. \* Added temporary logging to better identify which traffic was driving load, and worked with the customer to understand their usage pattern. \* Applied temporary protective rate-limiting on the affected endpoint as a safeguard. \* Shipped a permanent API optimization that removes the search-service dependency from this workflow-event-processing path, which reduced search load by about 95%. 5. Preventive Actions 1. Add duration-based alerting for sustained search-cluster CPU saturation, correlated with event-processing lag, so prolonged stress escalates automatically rather than depending on manual observation. 2. Validate search-cluster capacity headroom going forward, and evaluate whether capacity can now be safely reduced given the dependency has been removed from the hot path. 3. Decouple or bound similar dependencies: audit other workflow-event-processing paths for synchronous search-service calls and inventory similar dependencies elsewhere in the platform. 4. Add finer-grained rate limiting on the affected endpoint so that one customer's traffic pattern cannot affect other customers sharing the same infrastructure. 5. Define a clearer policy for applying protective traffic controls when a single customer's usage threatens shared-platform stability. ‌ ‌ \_We appreciate your patience while we worked through a complete resolution, and we remain committed to strengthening the resilience of our platform as a result of this incident.\_ From the builder: does your app depend on Harness? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=harness) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Harness. --- down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-7155f9d112), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.