Incident with Pull Requests: High percentage of 500s

What to do

• Check whether you use GitHub Pull Requests. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 6h 18m it lasted.

  Anything that failed between 2026-03-31 15:05 UTC and 2026-03-31 21:23 UTC may need a replay.

• Subscribe to GitHub's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=github works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from GitHub 11

1. investigatingMar 31, 2026, 3:05 PM UTC

   We are investigating reports of degraded performance for Pull Requests

   provider

2. investigatingMar 31, 2026, 3:06 PM UTC

   We are seeing a higher than average number of 500s due to timeouts across GitHub services. We have a potential mitigation in flight and are continuing to investigate.

   provider

3. investigatingMar 31, 2026, 3:39 PM UTC

   We are investigating increased 500 errors affecting GitHub services. You may experience intermittent failures when using Pull Requests and other features. We are actively working to identify and resolve the underlying cause.

   provider

4. investigatingMar 31, 2026, 4:15 PM UTC

   We are continuing to investigate increased 500 errors affecting GitHub services. You may experience intermittent failures when using Pull Requests and other features. We are actively working to identify and resolve the underlying cause.

   provider

5. investigatingMar 31, 2026, 4:35 PM UTC

   We are seeing recovery in latency and timeouts of requests related to pull requests, even though 500s are still elevated. While we are continuing to investigate, we are applying a mitigation and expect further recovery after it is applied.

   provider

6. investigatingMar 31, 2026, 5:16 PM UTC

   We identified an issue causing increased errors when accessing Pull Requests. The mitigation is being applied across our infrastructure and we will continue to provide updates as the mitigation rolls out.

   provider

7. investigatingMar 31, 2026, 6:42 PM UTC

   We continue to experience elevated error rates affecting Pull Requests. An earlier fix resolved one component of the issue, but some users may still encounter intermittent timeouts when viewing or interacting with pull requests. Our teams are actively investigating the remaining causes.

   provider

8. investigatingMar 31, 2026, 7:28 PM UTC

   Error rates remain elevated across multiple pull request endpoints. We are pursuing multiple potential mitigations.

   provider

9. investigatingMar 31, 2026, 9:12 PM UTC

   We continue to see a small subset of repositories experiencing timeouts and elevated latency in Pull Requests, affecting under 1% of requests.

   provider

10. monitoringMar 31, 2026, 9:16 PM UTC

    The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.

    provider

11. resolvedMar 31, 2026, 9:23 PM UTC

    On Monday March 31st, 2026, between 13:53 UTC and 21:23 UTC the Pull Requests service experienced elevated latency and failures. On average, the error rate was 0.15% and peaked at 0.28% of requests to the service. This was due to a change in garbage collection (GC) settings for a Go-based internal service that provides access to Git repository data. The changes caused more frequent GC activity and elevated CPU consumption on a subset of storage nodes, increasing latency and failure rates for some internal API operations. We mitigated the incident by reverting the GC changes. To prevent future incidents and improve time to detection and mitigation, we are instrumenting additional metrics and alerting for GC-related behavior, improving our visibility into other signals that could cause degraded impact of this type, and updating our best practices and standards for garbage collection in Go-based services.

    provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.