# AIDI Dashboards – Degraded Performance

- Provider: [Harness](https://down.guide/harness)
- Level: confirmed
- Impact: Degraded
- lasted 8h 22m
- started 2026-07-22 14:55 UTC
- Software Engineering Insights (SEI) /AI DLC Insights, Software Engineering Insights (SEI) / AI DLC Insights, Software Engineering Insights (SEI) / AI DLC Insights
- id: DWN-8c3c3f88e8
- Official page: https://stspg.io/kzwzjz0p0wcp

## Updates from Harness
### 2026-07-22 14:55 UTC: investigating
We are investigating an issue impacting AIDI dashboards. Users may experience increased load times or intermittent failures when accessing dashboards. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available.

### 2026-07-22 14:58 UTC: identified
The issue has been identified and a fix is being implemented.

### 2026-07-22 15:25 UTC: monitoring
A fix has been implemented and we are monitoring the results.

### 2026-07-22 23:16 UTC: resolved
This incident has been resolved.

### 2026-08-10 18:17 UTC: postmortem
Summary
Customers on Prod1, Prod2, and Prod3 clusters experienced intermittent widget load failures and increased load times when accessing AIDI 2.0 dashboards on July 22, 2026. Not all widgets were affected simultaneously the issue manifested as sporadic failures rather than a full outage.
No customer data was lost. SEI 1.0 customers were not impacted.
Root Cause
Over time, a routine database maintenance process failed to run on certain tables in our analytics database, causing those tables to accumulate a large volume of internal metadata used to track deleted records. When the database planned queries against these tables, it loaded all of this accumulated metadata into memory, causing memory usage on the affected nodes to spike repeatedly. These repeated spikes triggered an automatic safety mechanism that restarts a node when it detects excessive memory pressure, and the affected nodes began restarting in a loop as a result. This caused intermittent, degraded query performance on AIDI 2.0 dashboards for the duration of the incident.
Impact
Customers on Prod1, Prod2, and Prod3 clusters may have experienced intermittent widget load failures or increased load times on AIDI 2.0 dashboards.
Duration: July 22, 2026, 07:58 PDT – 16:16 PDT \\(~8 hours 18 minutes\\), with intermittent widget failures; system was restarted and under active monitoring from 08:25 PDT onward.
What was not impacted?
\* Data ingestion and processing
\* SEI 1.0 customers
\* Integrations and metadata flows
No customer data was lost.
Remediation
Upon identifying the issue, the affected database nodes were restarted at 08:25 PDT, which restored initial stability. We continued to monitor the system closely, and when intermittent degradation was still observed afterward, we applied several additional fixes:
\* Adjusted database configuration settings to limit the amount of memory used for processing accumulated metadata, and tuned query-planning settings to reduce memory pressure.
\* Ran cleanup jobs to reduce the backlog of accumulated metadata on the affected tables.
\* Increased capacity on the affected database nodes to provide additional headroom.
These changes progressively stabilized the system, and the incident was fully resolved at 16:16 PDT.
Action Items
To prevent recurrence, we are implementing the following:
1. We have upgraded the backend which includes underlying improvements that handle memory spikes caused by excessive delete files.
2. We have rolled out automated compaction jobs for newly introduced tables to prevent delete file accumulation going forward.

From the builder: does your app depend on Harness? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=harness) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Harness.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-8c3c3f88e8), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.