DNS in us-east-1 is degraded

What to do

• Check whether you use Trigger.dev in eu, eu-central-1, us, us-east-1. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2d 1h it lasted.

  Anything that failed between 2026-04-21 15:40 UTC and 2026-04-23 17:15 UTC may need a replay.

• Subscribe to Trigger.dev's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=trigger-dev works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Trigger.dev 3

1. updateApr 21, 2026, 3:40 PM UTC

   We are seeing some DNS resolution issues in us-east-1. OTel spans in the web dashboard are also affected. We are investigating the root cause. In the meantime, please consider switching to the eu-central-1 region while we investigate.

   provider

2. updateApr 21, 2026, 4:15 PM UTC

   The issue has been resolved and DNS service is back to normal. We'll continue investigation and share further details in this incident report.

   provider

3. resolvedApr 23, 2026, 5:15 PM UTC

   ## Postmortem: Intermittent DNS Resolution Failures (US region) **Date**: 2026-04-21 **Duration**: ~1 hour **Impact**: Intermittent DNS resolution failures for tasks running in our US region ### Summary Between approximately 15:50 and 16:50 UTC on 2026-04-21, some tasks running in our US region experienced intermittent DNS resolution failures when connecting to external services such as databases, object storage, and third-party APIs. The errors typically surfaced as `EAI_AGAIN` or transient name-resolution timeouts. Our EU region was unaffected. The platform itself remained available throughout, and the incident self-resolved before any manual intervention. ### What customers experienced Tasks in the US region that made outbound connections to external hostnames during the affected window could see intermittent resolution failures. Cluster-internal communication and task scheduling were not impacted. We did not observe degradation of the dashboard, API, or engine. ### Root cause Under elevated workload, an internal telemetry service hit its memory limit and restarted. When it came back up, a large number of running tasks simultaneously re-established their connections to it, triggering a burst of DNS lookups for a single internal hostname. A default DNS configuration on our pods amplified that burst by roughly 10x at the wire level, because each hostname resolution attempted several search-path variants, and each of those was duplicated across the IPv4 and IPv6 query families. The telemetry service then went into a short restart loop, so the reconnection surge repeated, keeping DNS resolution degraded across the window. Some external name-resolution attempts from customer task code ran long enough to hit their own timeouts and surfaced as `EAI_AGAIN`. During the incident, our cluster DNS resolver remained healthy by its own metrics. The failure was silent at the node-network layer and was only fully reconstructable from after-the-fact packet capture. This is the main reason we could not attribute the degradation in real time, and it is the primary gap we have closed in the days since. ### What we've done - **Further scaled the telemetry service** horizontally, so there's more capacity headroom under workload spikes, and a simultaneous all-replicas restart is less likely. - **Deployed a per-node DNS cache** across our worker fleet to absorb redundant lookup patterns at source, before they reach the cluster DNS resolver. - **Reduced DNS amplification at source** on platform-managed services, so a single hostname lookup produces one wire query instead of ten. - **Extended our monitoring** for network-level silent drops, per-upstream DNS resolver latency, and telemetry pipeline health. - **Shipped a suite of production alerts** calibrated against this incident's actual signature. A future incident of this class should be detected in minutes rather than reconstructed after the fact. ### Ongoing - A separate observation about how our cluster DNS resolver forwards queries under load is under continued investigation. It is not currently known to cause user-facing impact, but it is a gap in our understanding that we are closing. - We are re-evaluating DNS defaults on customer-facing tasks to remove the amplification pattern entirely, rather than just absorbing it with caching. We take incidents of this class seriously even when they self-resolve. The multi-day investigation was a deliberate choice to reconstruct the event with enough fidelity to fix the underlying observability gap, not just the immediate symptoms. We apologise for the disruption.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.