AIDI Dashboards – Degraded Performance

What to do

• Check whether you use Harness Software Engineering Insights (SEI) /AI DLC Insights, Software Engineering Insights (SEI) / AI DLC Insights, Software Engineering Insights (SEI) / AI DLC Insights. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 8h 22m it lasted.

  Anything that failed between 2026-07-22 14:55 UTC and 2026-07-22 23:16 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 5

1. investigatingJul 22, 2026, 2:55 PM UTC

   We are investigating an issue impacting AIDI dashboards. Users may experience increased load times or intermittent failures when accessing dashboards. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available.

   provider

2. identifiedJul 22, 2026, 2:58 PM UTC

   The issue has been identified and a fix is being implemented.

   provider

3. monitoringJul 22, 2026, 3:25 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

4. resolvedJul 22, 2026, 11:16 PM UTC

   This incident has been resolved.

   provider

5. postmortemAug 10, 2026, 6:17 PM UTC

   ## Summary Customers on Prod1, Prod2, and Prod3 clusters experienced intermittent widget load failures and increased load times when accessing AIDI 2.0 dashboards on July 22, 2026. Not all widgets were affected simultaneously the issue manifested as sporadic failures rather than a full outage. No customer data was lost. SEI 1.0 customers were not impacted. ## Root Cause Over time, a routine database maintenance process failed to run on certain tables in our analytics database, causing those tables to accumulate a large volume of internal metadata used to track deleted records. When the database planned queries against these tables, it loaded all of this accumulated metadata into memory, causing memory usage on the affected nodes to spike repeatedly. These repeated spikes triggered an automatic safety mechanism that restarts a node when it detects excessive memory pressure, and the affected nodes began restarting in a loop as a result. This caused intermittent, degraded query performance on AIDI 2.0 dashboards for the duration of the incident. ## Impact Customers on Prod1, Prod2, and Prod3 clusters may have experienced intermittent widget load failures or increased load times on AIDI 2.0 dashboards. **Duration:** July 22, 2026, 07:58 PDT – 16:16 PDT \(~8 hours 18 minutes\), with intermittent widget failures; system was restarted and under active monitoring from 08:25 PDT onward. ### What was not impacted? * Data ingestion and processing * SEI 1.0 customers * Integrations and metadata flows No customer data was lost. ## Remediation Upon identifying the issue, the affected database nodes were restarted at 08:25 PDT, which restored initial stability. We continued to monitor the system closely, and when intermittent degradation was still observed afterward, we applied several additional fixes: * Adjusted database configuration settings to limit the amount of memory used for processing accumulated metadata, and tuned query-planning settings to reduce memory pressure. * Ran cleanup jobs to reduce the backlog of accumulated metadata on the affected tables. * Increased capacity on the affected database nodes to provide additional headroom. These changes progressively stabilized the system, and the incident was fully resolved at 16:16 PDT. ## Action Items To prevent recurrence, we are implementing the following: 1. We have upgraded the backend which includes underlying improvements that handle memory spikes caused by excessive delete files. 2. We have rolled out automated compaction jobs for newly introduced tables to prevent delete file accumulation going forward.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.