- Level
- confirmed
- Worst impact
- Degraded
- Components
- US-Central-1 (Texas), US-East-1 (N. Virginia), US-East-2 (N. Virginia), US-West-1 (Oregon), CA-Central-1 (Toronto), EU-Central-1 (Amsterdam), EU-Central-2 (Frankfurt), EU-West-1 (London), EU-West-2 (Paris), EU-West-3 (London), EU-South-1 (Milan), AP-Northeast-1 (Tokyo), AP-Northeast-2 (Osaka), AP-Southeast-1 (Singapore), AP-Southeast-2 (Sydney), Wasabi Account Control API, Wasabi Account Control Manager Console, Wasabi Management Console
- Regions
- ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, apac, au, ca, ca-central-1, eu, eu-central-1, eu-central-2, eu-south-1, eu-west-1, eu-west-2, eu-west-3, us, us-central-1, us-east-1, us-east-2, us-west-1
- Down Index
- 41of 100
What to do
Check whether you use Wasabi US-Central-1 (Texas), US-East-1 (N. Virginia), US-East-2 (N. Virginia), US-West-1 (Oregon) and 14 more in ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 4h 26m it lasted.
Anything that failed between 2024-11-30 00:20 UTC and 2024-11-30 04:46 UTC may need a replay.Subscribe to Wasabi's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=wasabi works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Wasabi 6
- investigating
We are currently investigating reported network errors across all regions. Access to both Console and S3 services may return errors. We will update this page as we have more information.
provider
- investigating
We continue to investigate the issue. Some traffic continues to receive errors.
provider
- monitoring
We have isolated the issue and resolution is underway. We expect all regions except us-east-1 to be successful. There is still some level of error responses being seen in Ashburn and we continue to work on that. We will continue to monitor all regions and update their status as well as that of us-east-1 as we continue to make progress.
provider
- monitoring
The system has been restored to fully operational in all regions. We will populate the Postmortem section of this incident with more complete details as soon as possible.
provider
- resolved
This incident has been resolved.
provider
- postmortem
On 30 November 2024 from 2024-11-30 00:17 UTC to approximately 2024-11-30 03:00 UTC, Wasabi experienced an issue where client connection attempts to Wasabi Cloud Storage and the Web Console were impacted across all storage regions, resulting in all API calls to be returned as HTTP 5XX errors to clients. The cause behind this service degradation was due to an error within the internal messaging queue service responsible for taking client requests and routing them to our global database cluster. The internal messaging queue service failed to appropriately route these client requests across all nodes. Wasabi’s Engineering and Operations teams was able to mitigate this issue by manually configuring internal servers to route requests across multiple database instances, allowing the system to recover and respond to requests appropriately. Once this action was taken, which restored service, the teams then worked to correct the root cause by working to recover the internal messaging queue and resume the automated task of proper client request handling.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-3a58e38a57
Is Wasabi still down?
No, this incident was resolved 2024-11-30 04:46 UTC. For Wasabi's live state, see https://down.guide/wasabi.
How long did it last?
4h 26m, from 2024-11-30 00:20 UTC to 2024-11-30 04:46 UTC, by the times Wasabi posted.
What was affected?
Wasabi named US-Central-1 (Texas), US-East-1 (N. Virginia), US-East-2 (N. Virginia), US-West-1 (Oregon) and 14 more in ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2.
What caused it?
Wasabi hasn't published a root cause. If it does, it shows here.
Where does this come from?
Wasabi's official status page, https://status.wasabi.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.