Login errors in US region

What to do

• Check whether you use Keeper Keeper Web Vault (US), Keeper Admin Console (US) in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 19m it lasted.

  Anything that failed between 2025-02-10 17:30 UTC and 2025-02-10 17:49 UTC may need a replay.

• Subscribe to Keeper's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=keeper works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Keeper 3

1. identifiedFeb 10, 2025, 5:30 PM UTC

   We are aware of a high number of errors in the US region. The issue will be fixed shortly.

   provider

2. resolvedFeb 10, 2025, 5:49 PM UTC

   We have resolved the issue and plan to deploy a backend update shortly.

   provider

3. postmortemFeb 10, 2025, 9:53 PM UTC

   At 9:21AM PST we received notifications of API errors coming from our backend systems. The issue resolved itself without any required action at 9:31AM PST. During that 10 minute period, there was a database lock in the primary writer instance causing new login sessions to fail. Recently, Keeper upgraded all production RDS Aurora databases to MySQL 8.x. This updated database engine is sensitive to one particular SQL query that was causing large table scans to occur. The code changes and QA process was already under way when this issue occurred. The backend code change which addresses this issue passed QA at 12:48PM PST and we released the fix to prevent this issue from occurring again \(Release Jira ticket REL-4633\). We’re keeping a close eye and will issue additional updates as necessary.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.