Skip to main content
Duo Security

Admin functions impacted when accessing Users and AD Sync on certain deployments

Major outage Lasted 3h 35m · started · Duo Security's page

Level
confirmed
Worst impact
Major outage
Components
Admin Panel, Admin Panel, Admin Panel, SSO, SSO, SSO
Regions
us
Down Index
46of 100

What to do

  • Check whether you use Duo Security Admin Panel, Admin Panel, Admin Panel, SSO and 2 more in us. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 3h 35m it lasted.

    Anything that failed between 2025-08-26 08:08 UTC and 2025-08-26 11:43 UTC may need a replay.
  • Subscribe to Duo Security's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=duo works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Duo Security 6

  1. investigating

    We are currently investigating an issue impacting user management and Active Directory sync on selected deployments. Please refer to the status page for further updates.

    provider

  2. investigating

    We are continuing to investigate this issue.

    provider

  3. identified

    A fix has been deployed for the issue affecting User Management and SSO logins on Duo3, Duo47, and Duo57. We are monitoring performance and stability. Please refer to the status page for further updates.

    provider

  4. monitoring

    Following the implementation of the fix, we are seeing signs of recovery. We will continue to actively monitor the results to ensure the issue is fully resolved. Please refer to the status page for further updates.

    provider

  5. resolved

    The issue affecting User Management and SSO logins on Duo3, Duo47, and Duo57 has been fully resolved, and all services have been restored to normal functionality. We will publish a RCA as soon as it is available. You can check back here for the latest information or subscribe for updates.

    provider

  6. postmortem

    **Summary** A change in the D322 build triggered high database load by calling an unoptimized query. This resulted in disruptions across several deployments. The system was stabilized after recovery actions. **Timeline of Events** 08/26/2025 06:47:00 UTC - Alert fires that our service latency is above target.  08/26/2025 06:58:00 UTC - On-call engineer confirms that autoscaling is in progress.  08/26/2025 07:00:00 UTC - Separate alert fires that our service database CPU utilization is high, and requests backlog starts to grow.  08/26/2025 07:37:00 UTC - On-call engineer disables certain features of our service to alleviate load.  08/26/2025 08:08:00 UTC - Status page is published.  08/26/2025 08:45:00 UTC - Service database is scaled up.  08/26/2025 08:50:00 UTC - On-call engineer monitors recovery of systems. 08/26/2025 10:40:00 UTC - All features of our service are reenabled.  08/26/2025 11:44:00 UTC - Status page is updated to resolved.  **Details** Our D322 build started calling an unoptimized API endpoint on an internal system which led to high database load, latency and request timeouts. This degraded multiple deployments \(DUO3, DUO47, DUO57\). ‌ To alleviate load, certain features were disabled temporarily. However, because Duo Directory depends on this feature, customers received errors until it was re-enabled. At the same time, the database was scaled up vertically to allow for quicker processing of backlogged requests. ‌ Once the backlog was cleared, all features were reenabled and the incident was resolved. ‌ Deep analysis was conducted to determine why the service database CPU utilization was higher than expected. A newly introduced unoptimized query was found and fixed promptly. ‌ Furthermore, our teams have plans to implement more rigorous testing and review processes for new database queries that are added to our services. ‌ Also, our teams will work to improve our performance testing capabilities in the development environment to allow more accurate simulation of production load. This will aid us in being able to better detect unoptimized queries before they are released.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-223591d96e

Is Duo Security still down?

No, this incident was resolved 2025-08-26 11:43 UTC. For Duo Security's live state, see https://down.guide/duo.

How long did it last?

3h 35m, from 2025-08-26 08:08 UTC to 2025-08-26 11:43 UTC, by the times Duo Security posted.

What was affected?

Duo Security named Admin Panel, Admin Panel, Admin Panel, SSO and 2 more in us.

What caused it?

Duo Security hasn't published a root cause. If it does, it shows here.

Where does this come from?

Duo Security's official status page, https://status.duo.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.