Controller downtime

What to do

• Check whether you use Twingate Connector Heartbeat in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 39m it lasted.

  Anything that failed between 2021-08-10 19:45 UTC and 2021-08-10 21:24 UTC may need a replay.

• Subscribe to Twingate's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=twingate works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Twingate 4

1. monitoringAug 10, 2021, 8:00 PM UTC

   The Controller infrastucture was experiencing degraded availability. The issue began at 19:39 UTC and continued until 19:47 UTC. Our team is currently investigating the root cause of the issue, and we will post additional updates here.

   provider

2. monitoringAug 10, 2021, 8:11 PM UTC

   The Controller is currently fully available, and we are actively investigating the root cause of the issue.

   provider

3. resolvedAug 10, 2021, 9:24 PM UTC

   This incident has been resolved. We will be posting a post mortem description shortly.

   provider

4. postmortemAug 12, 2021, 11:37 PM UTC

   **Components impacted** Controller **Summary** The Controller was partially unavailable to service requests from approximately 19:39 to 19:46 UTC. The result was that during this period of time access to protected resources was limited, some existing connections were dropped, and new connections were refused. At the end of the period, normal service resumed with no remediation required. **Root cause** Leading up to the start of the incident was a planned maintenance period. The maintenance change propagated a configuration change across our Relay clusters. Due to human error that change was not applied in a sequential manner one cluster at a time but was released instead to all of our US clusters in parallel. Once the configuration changed was applied, it triggered reconnection requests from all active Clients and Connectors to our Relay infrastructure. As part of the reconnect process, Clients and Connectors needed to obtain new tokens from the Controller. At 19:39 UTC the spike of requests triggered our health check system, which incorrectly determined that the Controller was misbehaving and required restarting. The frequent Controller restarts had the unintended consequence that resulted in a decrease in service availability. **Corrective actions** As soon as the health-check system kicked in, the DevOps and on-call engineering teams started tracing down the issue. Logs and system metrics confirmed that except for health-check system, everything was performing well, so a decision was made to disable it. Seconds after disabling it, the system returned to a fully operation state. At 21:22 UTC a hot fix was deployed to the health-check system and it was enabled once again. Looking ahead, we plan to: 1. Only perform planned Relay maintenance operations that require connection migration outside of peak traffic hours. 2. Enforce a stricter limit of the number of parallel Relay cluster deployments. 3. Fix issues identified with our health check system and improve our performance and stress-testing to include more aggressive connection migration scenarios. 4. Update the Twingate status page immediately upon confirmation of an issue impacting customers.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.