# Network issues affecting all regions

- Provider: [Wasabi](https://down.guide/wasabi)
- Level: confirmed
- Impact: Operational
- lasted 4h 26m
- started 2024-11-30 00:20 UTC
- US-Central-1 (Texas), US-East-1 (N. Virginia), US-East-2 (N. Virginia) +15
- ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2
- id: DWN-3a58e38a57
- Official page: https://stspg.io/69414c0dzw4g

## Updates from Wasabi
### 2024-11-30 00:20 UTC: investigating
We are currently investigating reported network errors across all regions. Access to both Console and S3 services may return errors. We will update this page as we have more information.

### 2024-11-30 03:08 UTC: investigating
We continue to investigate the issue. Some traffic continues to receive errors.

### 2024-11-30 03:43 UTC: monitoring
We have isolated the issue and resolution is underway. We expect all regions except us-east-1 to be successful. There is still some level of error responses being seen in Ashburn and we continue to work on that. 
We will continue to monitor all regions and update their status as well as that of us-east-1 as we continue to make progress.

### 2024-11-30 04:45 UTC: monitoring
The system has been restored to fully operational in all regions. We will populate the Postmortem section of this incident with more complete details as soon as possible.

### 2024-11-30 04:46 UTC: resolved
This incident has been resolved.

### 2024-12-04 14:08 UTC: postmortem
On 30 November 2024 from 2024-11-30 00:17 UTC to approximately 2024-11-30 03:00 UTC, Wasabi experienced an issue where client connection attempts to Wasabi Cloud Storage and the Web Console were impacted across all storage regions, resulting in all API calls to be returned as HTTP 5XX errors to clients. The cause behind this service degradation was due to an error within the internal messaging queue service responsible for taking client requests and routing them to our global database cluster. The internal messaging queue service failed to appropriately route these client requests across all nodes. 
Wasabi’s Engineering and Operations teams was able to mitigate this issue by manually configuring internal servers to route requests across multiple database instances, allowing the system to recover and respond to requests appropriately. Once this action was taken, which restored service, the teams then worked to correct the root cause by working to recover the internal messaging queue and resume the automated task of proper client request handling.

From the builder: does your app depend on Wasabi? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=wasabi) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Wasabi.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-3a58e38a57), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.