Data ingestion is delayed on Traceable US production

What to do

• Check whether you use Harness US - app.traceable.ai / api.traceable.ai in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 44m it lasted.

  Anything that failed between 2026-08-19 13:22 UTC and 2026-08-19 16:07 UTC may need a replay.

• Subscribe to Harness's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=harness works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Harness 8

1. investigatingAug 19, 2026, 1:22 PM UTC

   We are currently investigating this issue.

   provider

2. investigatingAug 19, 2026, 2:03 PM UTC

   We are continuing to investigate this issue.

   provider

3. investigatingAug 19, 2026, 2:55 PM UTC

   We are continuing to investigate this issue.

   provider

4. identifiedAug 19, 2026, 3:13 PM UTC

   The issue has been identified and a fix is being implemented.

   provider

5. monitoringAug 19, 2026, 3:22 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

6. monitoringAug 19, 2026, 3:33 PM UTC

   We are continuing to monitor for any further issues.

   provider

7. resolvedAug 19, 2026, 4:07 PM UTC

   This incident has been resolved.

   provider

8. postmortemAug 26, 2026, 5:20 PM UTC

   **Summary** On 19 August 2026 between 12:35 and 17:29 UTC, the Harness Application Security service experienced a significant disruption affecting both the customer-facing console and the data ingestion pipeline in the SaaS Production and US1 regions. ‌ **Root Cause** The internal configuration service that supplies runtime settings to nearly every other component became overloaded and entered a repeated restart cycle. Because so many services depend on it, the effects were broad: console pages such as protection policies, posture views, activity logs, API inventory, and custom policy failed to load or timed out, and downstream processing stalled while waiting for configuration it could not obtain. # **Customer impact** | **Dimension** | **Detail** | | --- | --- | | Console \(UI\) impact | Multiple pages failed to load or timed out, including protection policies, posture event pages and posture views inside dashboards and insight pages, activity log queries, API inventory screens, custom policy, and sensitive-data views and widgets. | | Ingestion impact | Security telemetry processing degraded severely and, in some paths, stopped entirely. Consumer lag grew across normalisation, grouping, anomaly detection, generation, and related processing stages. | | Data loss | A subset of telemetry ingested during the disruption was permanently dropped. | ‌ **Mitigation** Several intermediate mitigations additional CPU and memory, relaxed health-check thresholds, a database restart, and a larger connection pool ameliorated the issue. Disabling the new feature in both affected regions restored throughput sharply and durably. The incident was resolved at 17:29 UTC. ‌ # **Preventive actions** The following actions are committed and tracked internally to completion. The feature that triggered this incident remains disabled and will not be re-enabled until the work below is complete and validated. | **Action** | | --- | | | | OPtimize the code by tuning parameters such as cache eviction and retention , evaluate cursor-based pagination for bulk rule retrieval as rule counts grow | | Add a purpose-built database index for the service-scoping access pattern | | Remediate pipeline recovery semantics so consumers replay safely after position-marker loss instead of skipping backlog | | Mandate staged rollout for configuration overrides that alter downstream request patterns: low-volume cluster, then mid-volume, then high-volume | | Add backpressure and concurrency protection to the configuration service: circuit breaking, bounded queues, and timeout isolation | | Enhance observability by Instrumenting more detailed metrics |

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.