- Level
- confirmed
- Worst impact
- Degraded
- Components
- pypi.org - CDN, pypi.org - Backends
- Regions
- us
- Down Index
- 24of 100
What to do
Check whether you use PyPI pypi.org - CDN, pypi.org - Backends in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 3h 42m it lasted.
Anything that failed between 2022-07-29 15:19 UTC and 2022-07-29 19:01 UTC may need a replay.Subscribe to PyPI's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=pypi works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from PyPI 5
- monitoring
In order to resolve an issue in our CDN cache caused by a misconfiguration of Surrogate-Keys, PyPI will need to issue a "Purge All" of our CDN to clear out lingering cached objects that are not accessible to purge individually. This purge is very likely to impact performance when accessing PyPI for some time. In order to avoid purges like this in the future, our backends have been configured with Surrogate-Keys that will at least allow us to purge specific endpoint types rather than the entire service.
provider
- monitoring
Purge All has been issued and we are monitoring as the backends attempt to meet the request volume. We anticipate that the service will continue to stabilize and reach steady state over the next 30-90 minutes.
provider
- monitoring
Our backends have begun to stabilize as the caches are refilled. During the peak of the backend traffic from our CDN we recognized a few bottlenecks in our database layer and have provisioned additional storage IOPs in order to reduce impact of surges in the future.
provider
- monitoring
Error rates have returned to baseline, and latencies continue to return to normal across our services. We will continue to monitor as the database upgrades complete and caches re-fill.
provider
- resolved
Our CDN hit rate and latencies have recovered, this event is now resolved.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-953e6d7ea4
Is PyPI still down?
No, this incident was resolved 2022-07-29 19:01 UTC. For PyPI's live state, see https://down.guide/pypi.
How long did it last?
3h 42m, from 2022-07-29 15:19 UTC to 2022-07-29 19:01 UTC, by the times PyPI posted.
What was affected?
PyPI named pypi.org - CDN, pypi.org - Backends in us.
What caused it?
PyPI hasn't published a root cause. If it does, it shows here.
Where does this come from?
PyPI's official status page, https://status.python.org, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.