Some users are unable to interact with the admin console

What to do

• Check whether you use 1Password Sign in, Sign in, Sign in, Sign in and 4 more. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 9m it lasted.

  Anything that failed between 2025-09-24 03:00 UTC and 2025-09-24 04:10 UTC may need a replay.

• Subscribe to 1Password's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=1password works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from 1Password 6

1. investigatingSep 24, 2025, 3:00 AM UTC

   We are actively investigating an issue where some users are encountering errors when interacting with the admin console, which leads to errors during sign-in.

   provider

2. investigatingSep 24, 2025, 3:36 AM UTC

   We are continuing to investigate the issue.

   provider

3. identifiedSep 24, 2025, 3:44 AM UTC

   Our engineering team has identified the issue and are working towards mitigating.

   provider

4. monitoringSep 24, 2025, 3:57 AM UTC

   Our engineering team has rolled out mitigation to remedy affected users. We are monitoring the results.

   provider

5. resolvedSep 24, 2025, 4:10 AM UTC

   This incident has been resolved.

   provider

6. postmortemOct 8, 2025, 8:45 PM UTC

   # Incident Postmortem - Some customers are unable to interact with the admin console **Date of Incident:** 2025-09-24 **Time of Incident \(UTC\):** 02:27 - 17:16 **Service\(s\) Affected:** Admin console, Sign in **Impact Duration:** 36:49 ## Summary Some customers with certain account configurations were placed on a blocklist and presented with a 403 error page after accessing the admin console. ## Impact on Customers * **Admin console:** Affected customers were presented with a 403 error page whenever they tried to interact with any of the admin console pages. * **Log in:** Affected customers were also unable to log in to the application. * **Number of Affected Customers \(approximate\):** 515 * **Geographic Regions Affected \(if applicable\):** All regions ## What Happened? * **Timeline of Events \(UTC\):** * Sep 24th 2:27am: Spike in application monitoring alerted engineers to increased rates of IP blocking * Sep 24th 3:00am: Cause identified as a change to requests in the application, which had been partially rolled out via a feature flag. * Sep 24th 4:03am: The feature flag was enabled to all customers which reduced the spike, but IP blocks continued throughout the day. * Sep 24th 10:03pm: Merged an application change to revert the change to prevent the issue reoccurring. * Sep 25th 5:03pm: The change was deployed with scheduled application release, error rate dropped off shortly after. * **Root Cause Analysis:** The issue was caused by GET requests to the Users API exceeding the URL length limit due to a recent change to append a list of UUIDs to the request parameters to resolve customer reported performance issues. * **Contributing factors:** * Requests were switched from GET to POST to prevent requests from exceeding the URL limit, however an issue with the feature flag configuration was causing UUIDs to be sent with the GET endpoint. * An underlying issue with the feature flag not resolving as expected in the application. ## How Was It Resolved? * **Mitigation Steps:** Customers were manually removed from the blocklist at multiple points in time as we evaluated the root cause and worked to patch the root issue. * **Resolution Steps:** The issue was mitigated by removing UUIDs at the API level if a GET request is used. Additional logging has been added to identify the root cause of the feature flag configuration issue. * **Verification of Resolution:** We monitored our server logs to ensure that we did not observe any additional GET requests to the affected URL. ## What We Are Doing to Prevent Future Incidents * **Audit additional admin console API requests:** We’re performing a sweep of admin console API requests to ensure the utilization of POST requests with highly parameterized URLs. * **Remove the feature flag misconfiguration:** We’re correcting the way the feature flag is configured to ensure consistent outcomes. ## Next Steps and Communication No action is required from our customers at this time. ‌ We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.