# Incident with multiple GitHub services

- Provider: [GitHub Copilot](https://down.guide/github-copilot)
- Level: confirmed
- Impact: Major outage
- lasted 1h 18m
- started 2026-04-23 16:12 UTC
- Webhooks, Actions, Copilot
- id: DWN-15ae1b035a
- Official page: https://stspg.io/1wm8c7d53454

## Updates from GitHub Copilot
### 2026-04-23 16:12 UTC: investigating
We are investigating reports of degraded availability for Copilot and Webhooks

### 2026-04-23 16:19 UTC: investigating
We are investigating multiple unavailable services.

### 2026-04-23 16:34 UTC: investigating
Actions is experiencing degraded performance. We are continuing to investigate.

### 2026-04-23 16:52 UTC: investigating
We have identified the root problem and are working on mitigation.

### 2026-04-23 17:03 UTC: investigating
The degradation affecting Actions and Copilot has been mitigated. We are monitoring to ensure stability.

### 2026-04-23 17:04 UTC: investigating
Many services are mitigated and are validating the remaining services.

### 2026-04-23 17:10 UTC: investigating
Webhooks is operating normally.

### 2026-04-23 17:30 UTC: resolved
On April 23, 2026, between 16:03 UTC and 17:27 UTC, multiple GitHub services experienced elevated error rates and degraded performance due to DNS resolution failures originating from our DNS infrastructure in our VA3 datacenter. Approximately 5–7% of overall traffic was affected during the impact window: 
- Webhooks: ~0.35% of API requests returned 5xx (peak ~0.39%). ~0.88% of requests exceeded 3s latency; at peak, \>3s responses represented ~10% of Webhooks API traffic. 
- Copilot Metrics: ~9% of Copilot Insights dashboard requests returned 5xx. 
- Copilot cloud agents: ~10% of cloud agent sessions were affected and failing. 
- Octoshift: 0.88% of active repo migrations failed and 79% saw elevated durations (avg. 5.2 min) during this period. 
- Git Operations: averaged 1.25% errors over the duration of the incident, with a peak of 2.07% errors. 
- Actions: Workflow run status updates experienced delays of up to ~8s over the duration of the incident window. 
Our DNS infrastructure in VA3 entered a degraded state and began intermittently returning NXDOMAIN responses and timing out on lookups for both internal service discovery and external endpoints. This caused a cascading impact across the dependent services listed above. 
We identified a specific load pattern under which our DNS resolvers began failing. The evidence points to a recently introduced traffic-balancing mechanism, rolled out progressively to support our growth, as the root cause. We have since reverted this change. 
We are immediately prioritizing investments in a more controlled rollout and validation process, including a dedicated environment to safely shadow production DNS traffic and detect these failure modes before they can affect production.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).