- Level
- confirmed
- Worst impact
- Degraded
- Regions
- us
- Down Index
- 12of 100
What to do
Check whether you use Keeper in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.
Anything that failed between 2025-03-11 04:40 UTC and 2025-03-11 00:00 UTC may need a replay.Subscribe to Keeper's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=keeper works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Keeper 2
- resolved
Between 5:15PM PST and 5:30PM PST, there were a high number of Vault Login errors.
provider
- postmortem
### **Incident Summary** The Keeper DevOps team was alerted to API errors in the US region. Upon investigation, we identified a high number of database deadlocks affecting performance. To mitigate the immediate impact, we restarted the affected writer instance. ### **Root Cause** Historically, similar issues were linked to specific queries impacting MySQL 8.x updates on the RDS Aurora platform. In this case, we discovered that a utility designed to monitor deadlocks and capture diagnostic data was inadvertently contributing to the problem. A known bug in the MySQL minor version used in production caused our data-gathering query—intended to analyze deadlocks—to trigger additional locking, exacerbating the issue. ### **Resolution & Mitigation** * We promptly disabled the problematic monitoring utility and applied database parameter optimizations in collaboration with the AWS team to enhance performance under peak load. * These changes were fully implemented within 24 hours, and no further excessive row locks have been observed. * Our DBA team is actively working with backend engineers to further optimize write operations for improved performance. * A backend code update with additional optimizations is scheduled for release the week of March 17. We appreciate your patience and will continue to monitor and enhance system performance to prevent similar issues in the future.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-a761a0a1a3
Is Keeper still down?
No, this incident was resolved 2025-03-11 00:00 UTC. For Keeper's live state, see https://down.guide/keeper.
How long did it last?
0m, from 2025-03-11 04:40 UTC to 2025-03-11 00:00 UTC, by the times Keeper posted.
What was affected?
Keeper named in us.
What caused it?
Keeper hasn't published a root cause. If it does, it shows here.
Where does this come from?
Keeper's official status page, https://statuspage.keeper.io, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.