System Errors in US-EAST-2

What to do

• Check whether you use Wasabi US-East-2 (N. Virginia) in us, us-east-2. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 5h 7m it lasted.

  Anything that failed between 2025-05-24 11:22 UTC and 2025-05-24 16:29 UTC may need a replay.

• Subscribe to Wasabi's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=wasabi works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Wasabi 5

1. investigatingMay 24, 2025, 11:22 AM UTC

   We are currently experiencing system errors in our US-EAST-2 region. Customers may experience elevated HTTP 5XX error responses when interacting with their Wasabi bucket(s). We will update this page as we have more information.

   provider

2. identifiedMay 24, 2025, 12:46 PM UTC

   The issue has been identified, and we are making appropriate changes to the system.

   provider

3. monitoringMay 24, 2025, 1:10 PM UTC

   We have made changes, traffic is operating normally, and we are monitoring the systems.

   provider

4. resolvedMay 24, 2025, 4:29 PM UTC

   This incident has been resolved. Please contact support@wasabi.com if you continue to see errors related to this incident.

   provider

5. postmortemJun 3, 2025, 3:44 AM UTC

   On 23 May 2025 at 23:00 UTC, Wasabi’s us-east-2 \(Manassas\) vault began seeing a large increase in S3 API traffic, causing regional resources to become strained with excessive internal connections. This excessive, sustained increase in traffic to the region was outside of normal operating conditions. At 08:12 UTC on 24 May 2025, this high resource usage began to impact the vault, causing an increase in 5xx errors for API calls to S3 buckets. Upon investigation from our Operations Team, it was found that a small number of accounts were responsible for consuming a large portion of available resources within the region, causing the increase in HTTP 5XX error responses for other customers. Our Operations Team took action by limiting the available connections to these accounts, which allowed for the overall active connections to the region to drop substantially. By 08:32 UTC on 24 May 2025, the region was back to normal operation.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.