- Level
- confirmed
- Worst impact
- Major outage
- Components
- Connector Heartbeat
- Regions
- us
- Down Index
- 45of 100
What to do
Check whether you use Twingate Connector Heartbeat in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 1h 45m it lasted.
Anything that failed between 2021-07-29 15:15 UTC and 2021-07-29 17:00 UTC may need a replay.Subscribe to Twingate's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=twingate works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Twingate 5
- identified
We've identified the issue, which is being caused by excessive memory usage on our infrastructure.
provider
- monitoring
We're resolved the immediate issue by adding additional processing capacity and increasing memory limits on our controller infrastructructure.
provider
- monitoring
The system is now confirmed as fully operational. We are working on an incident report and taking steps to ensure that this issue will not happen in the future.
provider
- resolved
This incident has been resolved.
provider
- postmortem
**Components impacted** Controller **Summary** The controller was unavailable to service new authentication requests from approximately 15:17 to 15:19 UTC. The result was that during this period of time, new connection requests were rejected. Existing connections were not impacted. At the end of the outage period, normal service resumed with no remediation required. **Root cause** Leading up to the start of the outage period, automated monitoring alerted us to spikes in memory usage. At approximately 15:16 UTC we introduced a change to our cluster that was intended to increase memory availability. At approximately 15:17 UTC as this change was rolled out, it had the unintended consequence that resulted in a decrease in service availability, with the resulting rejection of most requests. **Corrective actions** At 15:18 UTC, seeing the decrease in service availability, we reverted the change and simultaneously made additional hardware available to the cluster. Normal service resumed approximately 45 seconds later as the change propagated. Looking ahead, we plan to: 1. Investigate introducing decoupling between inbound requests and our backend as the likely cause of the memory spikes that triggered the change that caused the outage.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-db393c871e
Is Twingate still down?
No, this incident was resolved 2021-07-29 17:00 UTC. For Twingate's live state, see https://down.guide/twingate.
How long did it last?
1h 45m, from 2021-07-29 15:15 UTC to 2021-07-29 17:00 UTC, by the times Twingate posted.
What was affected?
Twingate named Connector Heartbeat in us.
What caused it?
Twingate hasn't published a root cause. If it does, it shows here.
Where does this come from?
Twingate's official status page, https://status.twingate.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.