Recent DNS Activity Unavailable

What to do

• Check whether you use Twingate Admin Console. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 20m it lasted.

  Anything that failed between 2024-04-19 21:45 UTC and 2024-04-19 22:04 UTC may need a replay.

• Subscribe to Twingate's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=twingate works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Twingate 4

1. identifiedApr 19, 2024, 9:45 PM UTC

   Recent DNS Activity on admin is unavailable. We have identified the issue and working on a fix.

   provider

2. monitoringApr 19, 2024, 9:58 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. resolvedApr 19, 2024, 10:04 PM UTC

   This incident has been resolved.

   provider

4. postmortemApr 25, 2024, 2:44 PM UTC

   **Components impacted** Management: Admin Console **Summary** On April 20, 2024, between 5:32 GMT and 6:57 GMT, Recent DNS Activity on the Admin Console became unavailable. Shortly after the incident began, the Twingate on-call team received alerts regarding abnormal database activity. Workers on the clusters that manages DNS filtering logs starting seeing errors from the logs API, leading to excessive retries and database writes. To mitigate the issue, the DNS Log Streaming workers were temporarily disabled. The root cause was identified as a malfunction in the DNS Filtering Log API caused by a problematic dependency upgrade. Consequently, viewing DNS filtering logs and analytics in the Admin Console was temporarily unavailable. A rollback of the update was issued, and normal operations were restored at 6:57 GMT after which DNS filtering logs and analytics were available in the Admin Console. **Root cause** The DNS Filtering Log API went down due to a bad dependency upgrade. **Corrective actions** Already completed: * Rectified Admin Console's infinite retry logic by enhancing the retrieval of DNS activity logs during error states. * Optimized DNS Log Streaming retry and database write procedures to reduce unnecessary operations when no events are returned from the DNS Filtering API Short-term: * Improve the dependency upgrade process for the DNS FIltering API

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.