- Level
- confirmed
- Worst impact
- Major outage
- Components
- Kernels
- Down Index
- 47of 100
What to do
Check whether you use Hex Kernels. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 4h 32m it lasted.
Anything that failed between 2025-05-13 14:38 UTC and 2025-05-13 19:10 UTC may need a replay.Subscribe to Hex's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=hex works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Hex 7
- investigating
We are currently investigating this issue.
provider
- investigating
We are continuing to investigate this issue.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- identified
Some users are still experiencing issues as we implement a fix.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- resolved
This incident has been resolved.
provider
- postmortem
After fully analyzing the timeline of events, **we’ve confirmed that the root cause of the incident was an issue in the AWS EBS storage system backing our primary database replica**. The issue began on May 12 11:00 PM PDT according to AWS, and the effect was gradual degradation of performance on the replica, eventually leading to a cascade where even our primary database got backpressured on critical write operations, which triggered the instability causing the incident. The AWS issue was resolved May 13 9:32 AM PDT, after we pushed a database configuration change that caused the EBS system software to reset. We then stabilized and cleaned up systems on our end before resolving the incident. While the root cause of this incident was on the AWS side, we are not satisfied that our detection and response were fast enough. We are investing in the following mechanisms to improve: * We have automated testing to monitor provisioning of kernels, but it is combined with a larger testing suite that makes the signal less immediately actionable. We will be extracting this out as its own monitor so we can more quickly identify and react. * Our database monitoring around latencies will be made more comprehensive, including the replica database, so we do not miss this early signal. * We are improving our incident runbook to cover some of these checks earlier in the process so we zero in on the root cause more quickly.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-09e0520411
Is Hex still down?
No, this incident was resolved 2025-05-13 19:10 UTC. For Hex's live state, see https://down.guide/hex.
How long did it last?
4h 32m, from 2025-05-13 14:38 UTC to 2025-05-13 19:10 UTC, by the times Hex posted.
What was affected?
Hex named Kernels.
What caused it?
Hex hasn't published a root cause. If it does, it shows here.
Where does this come from?
Hex's official status page, https://status.hex.tech, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.