- Level
- confirmed
- Worst impact
- Major outage
- Down Index
- 45of 100
What to do
Check whether your product depends on Trigger.dev. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 1d 1h it lasted.
Anything that failed between 2026-06-30 09:40 UTC and 2026-07-01 11:22 UTC may need a replay.Subscribe to Trigger.dev's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=trigger-dev works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Trigger.dev 11
- update
Runs are executing normally. The replication of runs to ClickHouse which powers the Runs list are impacted. We're investigating.
provider
- update
The `runs.list()` SDK function and Runs dashboard page are degraded. This is due to slow replication into our ClickHouse cluster. We are investigating.
provider
- update
Spans and logs are also degraded. This is due to slow replication into our ClickHouse cluster. We are investigating.
provider
- update
We believe slow ClickHouse replication is due to long running merges with deeply nested output objects. We have vertically scaled and are working with ClickHouse support on mitigations.
provider
- update
The root cause has been identified and remediation work is taking place
provider
- update
Runs are triggering and executing normally. The runs list (runs.list API + dashboard) is slow and may be delayed. Run logs and spans may be delayed. Cause identified, remediation in progress. Next update in 30 mins.
provider
- update
Runs are triggering and executing normally. The runs list (runs.list API + dashboard) performance is now improving. Run logs and spans are also improving. We're actively monitoring the rollout of the remediation. Next update in 30 mins.
provider
- update
We're continuing to monitor API response times, which are steadily improving. Our mitigation efforts remain ongoing.
provider
- update
First, the important part: your runs are safe. Runs execute and are stored in Postgres as normal. This issue is isolated to ClickHouse, which we use as a read-optimised view store for querying runs. Triggering, execution, and storage have been unaffected throughout. This is a visibility issue, not a data loss one. The fix is downgrading ClickHouse to 25.12. That rollback runs at the ClickHouse Cloud infrastructure level, so we're coordinating with them to apply it. It's agreed and queued. ## What you're seeing While the view was degraded, the runs list, filtering, bulk actions, the query API, and all error monitoring and alerting were slow or unavailable, and dashboard charts, traces, logs, and span detail were blank or incomplete for affected runs. Run execution, billing, and your Postgres-stored run data were unaffected throughout. - Dashboard and API queries against run data are slow or failing. There was a window this morning where these reads were down entirely. - A small percentage of runs created since June 17 aren't yet in the correct state in the view. The runs ran and are stored. The view of them is catching up. - Run logs and spans have recovered. Run views are still catching up. ## What's causing it ClickHouse moved from 25.12 to 26.2 on June 17. The new version added a cap on JSON column complexity (`input_format_binary_max_type_complexity`, defaulting to 1000), measured across the combined outputs in each batched insert. Once a batch went over, those inserts stopped landing in the view. Then this morning, June 30, memory and CPU on the instance climbed sharply as the backlog of unmerged data caught up with it, driving reads down until it ran out of memory and caused today's outage. The 25.12 rollback resolves the root cause. ## What we've done so far - Cleared the data that couldn't merge so healthy merges can continue. Run logs and spans recovered quickly; run views are slower and still catching up. - Scaled the cluster horizontally to add headroom. - Lined up the 25.12 rollback with ClickHouse Cloud.
provider
- update
This incident has been resolved. We are continuing to monitor and a post-mortem will be posted soon.
provider
- resolved
Update: the full postmortem is now published: https://trigger.dev/blog/incident-report-jun-30-2026
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-fcbb10c654
Is Trigger.dev still down?
No, this incident was resolved 2026-07-01 11:22 UTC. For Trigger.dev's live state, see https://down.guide/trigger-dev.
How long did it last?
1d 1h, from 2026-06-30 09:40 UTC to 2026-07-01 11:22 UTC, by the times Trigger.dev posted.
What caused it?
Trigger.dev hasn't published a root cause. If it does, it shows here.
Where does this come from?
Trigger.dev's official status page, https://status.trigger.dev, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.