Incident with multiple GitHub services

What to do

• Check whether you use GitHub Webhooks, Actions, Copilot. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 18m it lasted.

  Anything that failed between 2026-04-23 16:12 UTC and 2026-04-23 17:30 UTC may need a replay.

• Subscribe to GitHub's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=github works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from GitHub 8

1. investigatingApr 23, 2026, 4:12 PM UTC

   We are investigating reports of degraded availability for Copilot and Webhooks

   provider

2. investigatingApr 23, 2026, 4:19 PM UTC

   We are investigating multiple unavailable services.

   provider

3. investigatingApr 23, 2026, 4:34 PM UTC

   Actions is experiencing degraded performance. We are continuing to investigate.

   provider

4. investigatingApr 23, 2026, 4:52 PM UTC

   We have identified the root problem and are working on mitigation.

   provider

5. investigatingApr 23, 2026, 5:03 PM UTC

   The degradation affecting Actions and Copilot has been mitigated. We are monitoring to ensure stability.

   provider

6. investigatingApr 23, 2026, 5:04 PM UTC

   Many services are mitigated and are validating the remaining services.

   provider

7. investigatingApr 23, 2026, 5:10 PM UTC

   Webhooks is operating normally.

   provider

8. resolvedApr 23, 2026, 5:30 PM UTC

   On April 23, 2026, between 16:03 UTC and 17:27 UTC, multiple GitHub services experienced elevated error rates and degraded performance due to DNS resolution failures originating from our DNS infrastructure in our VA3 datacenter. Approximately 5–7% of overall traffic was affected during the impact window: - Webhooks: ~0.35% of API requests returned 5xx (peak ~0.39%). ~0.88% of requests exceeded 3s latency; at peak, >3s responses represented ~10% of Webhooks API traffic. - Copilot Metrics: ~9% of Copilot Insights dashboard requests returned 5xx. - Copilot cloud agents: ~10% of cloud agent sessions were affected and failing. - Octoshift: 0.88% of active repo migrations failed and 79% saw elevated durations (avg. 5.2 min) during this period. - Git Operations: averaged 1.25% errors over the duration of the incident, with a peak of 2.07% errors. - Actions: Workflow run status updates experienced delays of up to ~8s over the duration of the incident window. Our DNS infrastructure in VA3 entered a degraded state and began intermittently returning NXDOMAIN responses and timing out on lookups for both internal service discovery and external endpoints. This caused a cascading impact across the dependent services listed above. We identified a specific load pattern under which our DNS resolvers began failing. The evidence points to a recently introduced traffic-balancing mechanism, rolled out progressively to support our growth, as the root cause. We have since reverted this change. We are immediately prioritizing investments in a more controlled rollout and validation process, including a dedicated environment to safely shadow production DNS traffic and detect these failure modes before they can affect production.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.