Power Issues in the EU-CENTRAL-1 region

What to do

• Check whether you use Wasabi EU-Central-1 (Amsterdam) in eu, eu-central-1. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 9h 5m it lasted.

  Anything that failed between 2024-10-31 21:44 UTC and 2024-11-01 06:49 UTC may need a replay.

• Subscribe to Wasabi's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=wasabi works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Wasabi 6

1. investigatingOct 31, 2024, 9:44 PM UTC

   We have reports of power issues in our EU-CENTRAL-1 region and some users may be experiencing errors when interfacing with their Wasabi buckets hosted there. We will update our status page with more information as it becomes available.

   provider

2. investigatingOct 31, 2024, 10:21 PM UTC

   A facility power distribution issue is impacting several services in the eu-central-1 region. Direct Connect links are impacted as well as a subset of buckets. We continue to work with the facility to restore service.

   provider

3. identifiedOct 31, 2024, 11:15 PM UTC

   We have identified the issue and impacted resources. Team continue to work to restore service.

   provider

4. monitoringNov 1, 2024, 5:17 AM UTC

   The power distribution issue has been resolved and all affected resources are back in service. We are currently monitoring the status. If you see any error, please reach out to support@wasabi.com for assistance.

   provider

5. resolvedNov 1, 2024, 6:49 AM UTC

   The incident has been resolved. Please reach out to support@wasabi.com if you see any issues related to this incident.

   provider

6. postmortemNov 8, 2024, 1:24 AM UTC

   From 2024-10-31 at 21:10 UTC to 2024-11-01 at 04:00 UTC, we experienced issues with a subset of customer Buckets that were unavailable due to an unexpected power distribution issue while a planned power maintenance window was in progress at the EU-Central-1 region. The maintenance window required to run some servers on a single source and when the switch occurred, the power infrastructure could not keep up with the power draw from some of the infrastructure racks resulting in power down for those racks.    The impact resulted in failures in Reads/Write operations and Billing for impacted buckets. Our Operations team worked with the Equinix Datacenter team to restore the power and service availability was re-established as soon as the maintenance was over, and power was completely restored.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.