- Level
- confirmed
- Worst impact
- Partial outage
- Regions
- us
- Down Index
- 37of 100
What to do
Check whether you use Trigger.dev in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 21h 10m it lasted.
Anything that failed between 2025-12-01 14:08 UTC and 2025-12-02 11:18 UTC may need a replay.Subscribe to Trigger.dev's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=trigger-dev works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Trigger.dev 4
- update
We are currently having issues with our ingestion of open telemetry logs and spans after rolling out a fix for the issue that was happening over the weekend with clickhouse. We're investigating
provider
- update
otel ingestion has been operational since 1PM UTC but we are still finalizing a more permanent fix before we move the service from degraded to resolved.
provider
- update
we have pushed a fix which will prevent this ingestion issue from happening again. The issue came from a "Too many parts" error when inserting new data into clickhouse. This is a very common issue with clickhouse and was caused by a poorly designed partition key that caused inserts to create very small parts that would exhaust the clickhouse server's merge capacity, leading to merge failures. To mitigate this issue we initially created a new table that new runs would write their otel into that didn't suffer from this partition key issue. That worked fine, but because old runs were continuing to write to the old, poorly designed, table, the new table could not perform merges because of the previously stated issue. We immediately increased the number of clickhouse replicas which did buy us some merge headroom, allowing inserts to resume. But it was a race against the clock as we needed to ship a robust fix to the existing table before merge capacity was again exhausted. That fix has now been shipped and we're monitoring our clickhouse server's merge capacity which looks to be holding up.
provider
- resolved
We've published a full post-mortem on this incident here: https://trigger.dev/blog/clickhouse-too-many-parts-postmortem
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-ec78343db6
Is Trigger.dev still down?
No, this incident was resolved 2025-12-02 11:18 UTC. For Trigger.dev's live state, see https://down.guide/trigger-dev.
How long did it last?
21h 10m, from 2025-12-01 14:08 UTC to 2025-12-02 11:18 UTC, by the times Trigger.dev posted.
What was affected?
Trigger.dev named in us.
What caused it?
Trigger.dev hasn't published a root cause. If it does, it shows here.
Where does this come from?
Trigger.dev's official status page, https://status.trigger.dev, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.