# DNS in us-east-1 is degraded - Provider: [Trigger.dev](https://down.guide/trigger-dev) - Level: confirmed - Impact: Partial outage - lasted 2d 1h - started 2026-04-21 15:40 UTC - eu, eu-central-1, us, us-east-1 - id: DWN-5ed23ee016 - Official page: https://status.trigger.dev/incident/876588 ## Updates from Trigger.dev ### 2026-04-21 15:40 UTC: update We are seeing some DNS resolution issues in us-east-1. OTel spans in the web dashboard are also affected. We are investigating the root cause. In the meantime, please consider switching to the eu-central-1 region while we investigate. ### 2026-04-21 16:15 UTC: update The issue has been resolved and DNS service is back to normal. We'll continue investigation and share further details in this incident report. ### 2026-04-23 17:15 UTC: resolved ## Postmortem: Intermittent DNS Resolution Failures (US region) \*\*Date\*\*: 2026-04-21 \*\*Duration\*\*: ~1 hour \*\*Impact\*\*: Intermittent DNS resolution failures for tasks running in our US region ### Summary Between approximately 15:50 and 16:50 UTC on 2026-04-21, some tasks running in our US region experienced intermittent DNS resolution failures when connecting to external services such as databases, object storage, and third-party APIs. The errors typically surfaced as \`EAI\_AGAIN\` or transient name-resolution timeouts. Our EU region was unaffected. The platform itself remained available throughout, and the incident self-resolved before any manual intervention. ### What customers experienced Tasks in the US region that made outbound connections to external hostnames during the affected window could see intermittent resolution failures. Cluster-internal communication and task scheduling were not impacted. We did not observe degradation of the dashboard, API, or engine. ### Root cause Under elevated workload, an internal telemetry service hit its memory limit and restarted. When it came back up, a large number of running tasks simultaneously re-established their connections to it, triggering a burst of DNS lookups for a single internal hostname. A default DNS configuration on our pods amplified that burst by roughly 10x at the wire level, because each hostname resolution attempted several search-path variants, and each of those was duplicated across the IPv4 and IPv6 query families. The telemetry service then went into a short restart loop, so the reconnection surge repeated, keeping DNS resolution degraded across the window. Some external name-resolution attempts from customer task code ran long enough to hit their own timeouts and surfaced as \`EAI\_AGAIN\`. During the incident, our cluster DNS resolver remained healthy by its own metrics. The failure was silent at the node-network layer and was only fully reconstructable from after-the-fact packet capture. This is the main reason we could not attribute the degradation in real time, and it is the primary gap we have closed in the days since. ### What we've done - \*\*Further scaled the telemetry service\*\* horizontally, so there's more capacity headroom under workload spikes, and a simultaneous all-replicas restart is less likely. - \*\*Deployed a per-node DNS cache\*\* across our worker fleet to absorb redundant lookup patterns at source, before they reach the cluster DNS resolver. - \*\*Reduced DNS amplification at source\*\* on platform-managed services, so a single hostname lookup produces one wire query instead of ten. - \*\*Extended our monitoring\*\* for network-level silent drops, per-upstream DNS resolver latency, and telemetry pipeline health. - \*\*Shipped a suite of production alerts\*\* calibrated against this incident's actual signature. A future incident of this class should be detected in minutes rather than reconstructed after the fact. ### Ongoing - A separate observation about how our cluster DNS resolver forwards queries under load is under continued investigation. It is not currently known to cause user-facing impact, but it is a gap in our understanding that we are closing. - We are re-evaluating DNS defaults on customer-facing tasks to remove the amplification pattern entirely, rather than just absorbing it with caching. We take incidents of this class seriously even when they self-resolve. The multi-day investigation was a deliberate choice to reconstruct the event with enough fidelity to fix the underlying observability gap, not just the immediate symptoms. We apologise for the disruption. --- down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).