Skip to main content
GitHub

Incident with CodeQL, Webhooks, Notifications, and Slack Integration

Degraded Lasted 3h 5m · started · GitHub's page

Level
confirmed
Worst impact
Degraded
Components
Webhooks, Actions
Down Index
23of 100

What to do

  • Check whether you use GitHub Webhooks, Actions. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 3h 5m it lasted.

    Anything that failed between 2026-05-12 14:38 UTC and 2026-05-12 17:43 UTC may need a replay.
  • Subscribe to GitHub's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=github works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from GitHub 10

  1. investigating

    We are investigating reports of degraded performance for CodeQL

    provider

  2. investigating

    CodeQL actions are currently experiencing delays, which may result in those actions being stuck in a pending state or having failed due to a timeout.

    provider

  3. investigating

    We're continuing to investigate issues with CodeQL actions workflows. We're additionally seeing delays for notifications, webhooks, and the Slack integration.

    provider

  4. investigating

    Webhooks is experiencing degraded performance. We are continuing to investigate.

    provider

  5. investigating

    We've established that most delays are related to a queuing service and are working to scale out. Early signals from the scale-out are showing signs of recovery for some services. We'll provide an update when services are fully recovered.

    provider

  6. investigating

    Webhooks is operating normally.

    provider

  7. investigating

    Webhooks have fully recovered. Continuing to work on recovery for the other services.

    provider

  8. investigating

    CodeQL has fully recovered. We're continuing to work on recovery for the remaining impacted services.

    provider

  9. investigating

    All services have fully recovered.

    provider

  10. resolved

    On May 12, 2026, between 13:41 and 17:43 UTC, some services experienced delays in processing. For the Code Scanning service, 53% of check runs took over 15 minutes to complete. Additionally, notifications took an average of 22 minutes to be delivered and Slack integration webhooks took an average of 20 minutes to be delivered. The delays were caused by replication lag due to an internal database migration, resulting in insufficient worker capacity for our high rate of job enqueues. We mitigated the impact by scaling our processing workers to handle the increased load. All services returned to normal processing times after the mitigation was applied. We are working to create dedicated worker pools for some of our high usage shared queues to help prevent this in the future.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-289f82a4fa

Is GitHub still down?

No, this incident was resolved 2026-05-12 17:43 UTC. For GitHub's live state, see https://down.guide/github.

How long did it last?

3h 5m, from 2026-05-12 14:38 UTC to 2026-05-12 17:43 UTC, by the times GitHub posted.

What was affected?

GitHub named Webhooks, Actions.

What caused it?

GitHub hasn't published a root cause. If it does, it shows here.

Where does this come from?

GitHub's official status page, https://www.githubstatus.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.