Incident with Webhooks

What to do

• Check whether you use GitHub Webhooks. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 23m it lasted.

  Anything that failed between 2026-07-14 17:38 UTC and 2026-07-14 18:01 UTC may need a replay.

• Subscribe to GitHub's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=github works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from GitHub 4

1. investigatingJul 14, 2026, 5:38 PM UTC

   We are investigating reports of degraded performance for Webhooks

   provider

2. investigatingJul 14, 2026, 5:55 PM UTC

   Between 15:17 and 15:27 UTC, an ongoing deployment had an unintended side effect where webhook delivery states were not persisted in all cases, even when deliveries were accepted and processed. Customers may notice missing webhook deliveries in the UI, and these deliveries will not be retryable. Impact resolved automatically once the deployment completed.

   provider

3. monitoringJul 14, 2026, 5:55 PM UTC

   The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability.

   provider

4. resolvedJul 14, 2026, 6:01 PM UTC

   On July 14, 2026, between 15:17 and 15:37 UTC, a rollout to GitHub's internal webhook delivery pipeline caused a subset of webhook delivery records to not be written to our webhook deliveries store after being processed and delivered successfully. Affected deliveries would be missing from the webhook delivery UI and API and won’t be available for redelivery. The root cause was an uncoordinated rollout: a change to how delivery records are handed off between pipeline components was deployed before the upstream components producing those records were updated to match. While the rollout was in progress, affected records were silently skipped rather than persisted, with no automatic retry. The impact ended as soon as the rollout was completed. About 2.4M delivery records were skipped (approximately 4% of the 20-minute impact window, 0.04% of a typical 24-hour period). Importantly, 95% of these skipped deliveries reached customer endpoints successfully, only the record of the delivery is missing. Of the ~5% that failed to reach customer endpoints, only ~1.4% (5,463) map to webhooks that retried their deliveries in the past 28 days. To prevent recurrence, we are improving our automated detection of unsafe schema changes and tightening rollout coordination for changes that span multiple components in the pipeline.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.