Object storage provider outage

What to do

• Check whether you use PyPI files.pythonhosted.org - Files. Check your stack

  Drop a package.json, lockfile, Terraform or a list of domains at down.guide to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 4h 22m it lasted.

  Anything that failed between 2023-06-21 21:38 UTC and 2023-06-22 01:59 UTC may need a replay.

• Subscribe to PyPI's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=pypi works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from PyPI 6

1. investigatingJun 21, 2023, 9:38 PM UTC

   We are currently investigating an issue with our object storage provider.

   provider

2. identifiedJun 21, 2023, 10:46 PM UTC

   We have identified an issue that is impacting latencies for files.pythonhosted.org, resulting in timeouts for a portion of requests.

   provider

3. identifiedJun 21, 2023, 10:52 PM UTC

   We're in communication with the object storage provider and are waiting for them to rollout a fix.

   provider

4. identifiedJun 21, 2023, 10:57 PM UTC

   We are also implementing our own change to guard against excessive latencies from the provider.

   provider

5. monitoringJun 21, 2023, 11:45 PM UTC

   Our mitigation has decreased overall latencies, but they remain elevated over normal operation. As our provider's service recovers we expect to see a return to baseline.

   provider

6. resolvedJun 22, 2023, 1:59 AM UTC

   This incident has been resolved.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.