# Controller downtime

- Provider: [Twingate](https://down.guide/providers/twingate)
- Confidence: Confirmed
- Impact: Major outage
- lasted 1h 45m
- started 2021-07-29 15:15 UTC
- Connector Heartbeat
- us
- Twingate states the impact was 2021-07-29 15:17 to 15:19 UTC (2m)
- id: DWN-db393c871e
- Official page: https://stspg.io/qnlyrpzf8d2w

## Root cause
> We've identified the issue, which is being caused by excessive memory usage on our infrastructure.

[Postmortem](https://stspg.io/qnlyrpzf8d2w)

## Updates from Twingate
### 2021-07-29 15:15 UTC: identified
We've identified the issue, which is being caused by excessive memory usage on our infrastructure.

### 2021-07-29 15:17 UTC: monitoring
We're resolved the immediate issue by adding additional processing capacity and increasing memory limits on our controller infrastructructure.

### 2021-07-29 16:00 UTC: monitoring
The system is now confirmed as fully operational.
We are working on an incident report and taking steps to ensure that this issue will not happen in the future.

### 2021-07-29 17:00 UTC: resolved
This incident has been resolved.

### 2021-07-30 00:57 UTC: postmortem
Components impacted
Controller
Summary
The controller was unavailable to service new authentication requests from approximately 15:17 to 15:19 UTC. The result was that during this period of time, new connection requests were rejected. Existing connections were not impacted. At the end of the outage period, normal service resumed with no remediation required.
Root cause
Leading up to the start of the outage period, automated monitoring alerted us to spikes in memory usage. At approximately 15:16 UTC we introduced a change to our cluster that was intended to increase memory availability. At approximately 15:17 UTC as this change was rolled out, it had the unintended consequence that resulted in a decrease in service availability, with the resulting rejection of most requests.
Corrective actions
At 15:18 UTC, seeing the decrease in service availability, we reverted the change and simultaneously made additional hardware available to the cluster. Normal service resumed approximately 45 seconds later as the change propagated.
Looking ahead, we plan to:
1. Investigate introducing decoupling between inbound requests and our backend as the likely cause of the memory spikes that triggered the change that caused the outage.

From the builder: does your app depend on Twingate? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=twingate) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Twingate.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-db393c871e), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.