Elevated API Errors

What to do

• Check whether you use 1Password in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2025-09-03 11:00 UTC and 2025-09-03 11:00 UTC may need a replay.

• Subscribe to 1Password's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=1password works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from 1Password 2

1. resolvedSep 3, 2025, 11:00 AM UTC

   This incident has been resolved.

   provider

2. postmortemSep 10, 2025, 9:59 PM UTC

   # Performance Degradation **Date of Incident:** 2025-09-03 **Time of Incident \(UTC\):** 11:06 - 12:07 **Service\(s\) Affected:** All APIs **Impact Duration:** 61 minutes ## Summary For 61 minutes on the morning of September 3rd, 2025, all 1Password APIs in the US/Global environment had degraded performance or returned an error for approximately 20% of requests. 92% of the impact was mitigated within 13 minutes at 11:19 by automation scaling up infrastructure. By 12:06 a manual restart of the remaining infrastructure completed mitigation. A permanent fix was implemented and deployed to prevent the issue from reoccurring. ## Impact on Customers * **APIs:** High latency, or a 500 Internal Server Error. * **Number of Affected Customers:** 20% of all requests returned errors for 13 minutes, 1% thereafter. * **Geographic Regions Affected \(if applicable\):** 1Password USA/Global ## What Happened? * **Timeline of Events \(UTC\):** * 11:05: A customer started a stream of an unusually high volume of requests to an API with sub-optimal performance. * 11:06: Some servers started consuming abnormally high memory, causing slow response times and high error rates. * 11:19: Automation scaled up infrastructure to service additional load * 11:30: Increased errors trigger escalation, on-call engineer begins investigation * 11:51: Engineers declare an incident and alert response teams * 12:02: Response team begins restarting affected servers. * 12:07: All servers completed restarts, and error rates returned to normal levels * **Root Cause Analysis:** A poorly performing cache operation was triggered repeatedly in a short period of time across multiple servers, leading directly to greatly delayed responses. ## How Was It Resolved? * **Mitigation Steps:** Automatic instance scaling restored over 98% of operational capacity after 13 minutes. Full capacity was restored through manual intervention * **Resolution Steps:** We refactored the poorly performing query. * **Verification of Resolution:** We tested the affected API to confirm refactoring of query produced the desired performance improvement. We deployed the fix and monitored it for 24 hours to assert the issue was resolved. ## What We Are Doing to Prevent Future Incidents * We are auditing services for sub-optimal query performance. ## Next Steps and Communication * No action is required from our customers at this time. ‌ We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.