Skip to main content
JumpCloud

Increased Error Rates Affecting Multiple Platform Services

Partial outage Lasted 2h 38m · started · JumpCloud's page

Level
confirmed
Worst impact
Partial outage
Components
Admin Console - US Region, LDAP - US Region, RADIUS - US Region, User Console - US Region, TOTP / MFA / JumpCloud Protect - US Region, SSO - US Region
Regions
us
Down Index
34of 100

What to do

  • Check whether you use JumpCloud Admin Console - US Region, LDAP - US Region, RADIUS - US Region, User Console - US Region and 2 more in us. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 2h 38m it lasted.

    Anything that failed between 2025-11-04 11:40 UTC and 2025-11-04 14:19 UTC may need a replay.
  • Subscribe to JumpCloud's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=jumpcloud works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from JumpCloud 8

  1. investigating

    We are seeing issues with SSO authentication. We are investigating this currently and will update within 1 hour

    provider

  2. investigating

    We are continuing to investigate this issue.

    provider

  3. investigating

    We are continuing to investigate this issue.

    provider

  4. identified

    We have identified an issue that is causing intermittent login issues with the JumpCloud User Portal and Admin Portal. We have also identified issues with JumpCloud MFA, LDAP, RADIUS, and authentication with SSO. We are working on implementing a fix and will provide another update as soon as possible.

    provider

  5. identified

    We continue to see intermittent issues with accessing the JumpCloud User and Admin Portal, MFA, LDAP, RADIUS, and SSO. During this time access attempts to LDAP and RADIUS are only impacted if a user is authenticating with MFA. Our team is working on implementing a fix and will provide another update as quickly as possible.

    provider

  6. monitoring

    We have implemented a fix and users should now be able to access the User and Admin Console, MFA, LDAP, RADIUS, and SSO without issue.  We will continue to monitor the results of the fix.

    provider

  7. resolved

    Services have been fully restored and this incident has been resolved. We will provide a formal postmortem as a follow up.

    provider

  8. postmortem

    ![](https://jumpcloud.com/wp-content/themes/jumpcloud/assets/images/jumpcloud-press-kit/logos/02-jc-richblack-tm.png) ‌ **Date**: Nov 7, 2025 **Date of Incident:** Nov 4, 2025 **Description**: RCA for Auth Database Degradation ‌ **Summary:** On November 4, 2025, a number of customers experienced intermittent failures, timeouts and increased latency when attempting to authenticate to multiple JumpCloud Services, including consoles, LDAP, RADIUS and SAML, or use Multi-Factor Authentication. ‌ **Root Cause:** The incident was triggered by an issue in the deployment process involving a database schema change and a subsequent application code release. During this deployment, a planned database change unintentionally removed several database indexes required by the existing application code. The sequence of failure was as follows: 1. Deployment Order Error: The database schema change \(which removed necessary indexes\) was applied to the production database before the new application code \(which did not require those indexes\) was deployed. 2. Performance Collapse: The existing, high-volume authentication code \(used for functions like TOTP and push authentication\) was forced to run against the now-inefficient database structure. Queries that normally took milliseconds suddenly took several seconds. 3. Connection Exhaustion: These slow queries held database connections open for extended periods, quickly overwhelming the database server's available connection pool. 4. Full Outage: With no available connections, the main authentication API could not communicate with the database, leading to 100% CPU utilization on the database server and triggering the intermittent timeouts and failures experienced by our customers. ‌ **Why Testing Did Not Catch This:** The issue was not identified during testing in our Development or Staging environments due to insufficient Load Simulation. The resource consumption issues and connection exhaustion only manifest under the extreme pressure of peak production traffic volume. The simulated load profiles in our lower environments were not sufficient to expose this specific failure mode. ‌ **Corrective Actions / Risk Mitigation:** 1. Mandatory schema change review - All database schema changes must now undergo an additional level of review to explicitly assess index dependencies and impact. 2. New deployment phasing - We are implementing new tools and checks to enforce that application code dependent on a schema change is deployed before a database change is executed. 3. Enhance alerting -  We are implementing new monitors and alerts specifically for the Auth-API's database connection pool health and CPU utilization. 4. Enhanced load testing - We are revisiting the load profiles used in our staging environments looking for opportunities to more accurately simulate peak production traffic.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-f758f4f27b

Is JumpCloud still down?

No, this incident was resolved 2025-11-04 14:19 UTC. For JumpCloud's live state, see https://down.guide/jumpcloud.

How long did it last?

2h 38m, from 2025-11-04 11:40 UTC to 2025-11-04 14:19 UTC, by the times JumpCloud posted.

What was affected?

JumpCloud named Admin Console - US Region, LDAP - US Region, RADIUS - US Region, User Console - US Region and 2 more in us.

What caused it?

JumpCloud hasn't published a root cause. If it does, it shows here.

Where does this come from?

JumpCloud's official status page, https://status.jumpcloud.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.