Controller downtime

What to do

• Check whether you use Twingate Connector Heartbeat in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 11h 52m it lasted.

  Anything that failed between 2021-11-05 18:37 UTC and 2021-11-06 06:30 UTC may need a replay.

• Subscribe to Twingate's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=twingate works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Twingate 7

1. investigatingNov 5, 2021, 6:37 PM UTC

   We are looking into it we will provide more info as soon as we have it

   provider

2. investigatingNov 5, 2021, 6:49 PM UTC

   We are continuing to investigate this issue.

   provider

3. investigatingNov 5, 2021, 6:53 PM UTC

   We are back and operational now. We are still investigating the root cause.

   provider

4. investigatingNov 5, 2021, 6:54 PM UTC

   We are continuing to investigate this issue.

   provider

5. monitoringNov 5, 2021, 9:05 PM UTC

   We are still investigating the root cause of the incident. We didn't find any issue on our side, and we are working with our cloud provider support team to investigate the matter further.

   provider

6. resolvedNov 6, 2021, 6:30 AM UTC

   This incident has been resolved.

   provider

7. postmortemNov 9, 2021, 4:15 AM UTC

   **Components impacted** Relay Controller **Summary** A physical hardware failure occurred in a node within one of our Eastern US Relay clusters at approximately 18:19 UTC. Connectors and Clients attached to this node automatically failed over to a new node. This failover process resulted in a partial outage of the Controller, which was partially available to service requests from approximately 18:21 to 18:40 UTC. At the end of the period, normal service resumed with no remediation required. **Root cause** A physical hardware failure occurred in a single node within one of our Eastern US Relay clusters. Although the hardware was swapped out automatically by our service provider, this resulted in all Connectors attached to this particular Relay node to automatically failover to a new Relay node, resulting in a flood of connection requests. This process proceeded normally, however the volume of connection requests was sufficient in this particular instance to temporarily prevent the Controller from accepting new connection requests. This in turn resulted in additional reconnection requests, exacerbating the original problem. **Corrective actions** As soon as we received monitoring alerts, the DevOps and on-call engineering teams started triaging the issue. Additional nodes were started to handle the spike in connection requests and the system was monitored as the request rate recovered and the system was brought back to a normal running state at 18:40 UTC. Looking ahead, we have already or plan to: 1. Add additional nodes and increase memory limits across the board to serve as an additional buffer for failover-based connection spikes. 2. Make changes to our heartbeat monitoring logic to increase overall resilience during transient traffic peaks. 3. Introduce changes to the Connector logic to maintain connections to multiple Relay nodes at all times, resulting in a flatter spike in failover re-connection requests. 4. Introduce additional resiliency in token issuance to prevent temporary spikes in connection requests from influencing otherwise healthy Clients and Connectors.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.