Incident with Webhooks

What to do

• Check whether you use GitHub Webhooks. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 13m it lasted.

  Anything that failed between 2026-03-09 15:50 UTC and 2026-03-09 17:03 UTC may need a replay.

• Subscribe to GitHub's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=github works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from GitHub 4

1. investigatingMar 9, 2026, 3:50 PM UTC

   We are investigating reports of degraded performance for Webhooks

   provider

2. investigatingMar 9, 2026, 3:56 PM UTC

   We are experiencing latency on the API and UI endpoints. We are working to resolve the issue.

   provider

3. investigatingMar 9, 2026, 5:03 PM UTC

   Webhooks is operating normally.

   provider

4. resolvedMar 9, 2026, 5:03 PM UTC

   On March 9, 2026, between 15:03 and 20:52 UTC, the Webhooks API experienced was degraded, resulted in higher average latency on requests and in certain cases error responses. Approximately 0.6% of total requests exceeded the normal latency threshold of 3s, while 0.4% of requests resulted in 500 errors. At peak, 2.0% experienced latency greater than 3 seconds and 2.8% of requests returned 500 errors. The issue was caused by a noisy actor that led to resource contention on the Webhooks API service. We mitigated the issue initially by increasing CPU resources for the Webhooks API service, and ultimately applied lower rate limiting thresholds to the noisy actor to prevent further impact to other users. We are working to improve monitoring to more quickly ascertain noisy traffic and will continue to improve our rate-limiting mechanisms to help prevent similar issues in the future.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.