# Most routes 5xx in the US region \[resolved\]

- Provider: [Svix](https://down.guide/providers/svix)
- Confidence: Confirmed
- Impact: Partial outage
- lasted 48m
- started 2023-03-11 20:35 UTC
- API
- us
- id: DWN-8ade3d42f8
- Official page: https://stspg.io/jxyqtbd7b2mz

## Root cause
> There were no deploys today (Saturday), so it's not due to any change on our end, and the activity doesn't look unusual.

[Postmortem](https://stspg.io/jxyqtbd7b2mz)

## Updates from Svix
### 2023-03-11 19:38 UTC: investigating
We are currently investigating this issue.

### 2023-03-11 20:27 UTC: investigating
Still investigating.

### 2023-03-11 20:35 UTC: investigating
We are still investigating, but we put mitigating changes in place.

### 2023-03-11 20:42 UTC: monitoring
The service is back up. We are still monitoring, but everything is operational.

### 2023-03-11 21:23 UTC: resolved
The issue has been resolved, though we are still trying to locate the root cause.
There were no deploys today (Saturday), so it's not due to any change on our end, and the activity doesn't look unusual.
For whatever reason our API workers went from ~10% utilization (normal) to 100% in a short span of time. We are investigating with AWS.
Update: after investigating with AWS for a few hours, neither they nor us are able to understand the reason for the memory usage jump though their and our metrics don't indicate any change in load, underlying systems, or anything like that. They've indicated that OOM can happen even if there hasn't been any indication to that in the AWS metrics. 
We are still investigating.

### 2023-03-12 21:36 UTC: postmortem
We are still investigating, but here is our more complete update regarding this incident:
https://www.svix.com/blog/we-had-a-partial-outage/

From the builder: does your app depend on Svix? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=svix) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Svix.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-8ade3d42f8), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.