- Level
- confirmed
- Worst impact
- Major outage
- Components
- Main site
- Down Index
- 44of 100
What to do
Check whether you use Hex Main site. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 1h 26m it lasted.
Anything that failed between 2025-08-20 17:00 UTC and 2025-08-20 18:26 UTC may need a replay.Subscribe to Hex's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=hex works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Hex 7
- investigating
We are currently investigating this issue.
provider
- investigating
We are continuing to investigate this issue.
provider
- investigating
We are continuing to investigate this issue.
provider
- identified
The issue has been identified and a fix is being implemented.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- resolved
This incident has been resolved.
provider
- postmortem
After fully analyzing the timeline of events, **we've confirmed that the root cause was a cascading failure triggered by the combination of a feature flag, deployment timing during peak traffic, and an inefficiency in our database connection management**. The issue began August 19 at 4:03 PM PDT, when a feature flag was updated to have clients query the server more frequently to eliminate a rare display issue in the client. It escalated August 20 at 10:00 AM PDT when we did a production deployment during peak traffic to add necessary support for a new feature release. This combination created a thundering herd problem where client reconnection attempts exponentially increased system pressure, saturating our database connections and triggering system-wide failures. Service was restored for most users at 10:52 PM PDT after manual intervention, and then fully restored at 11:26 AM EDT for all users after resetting backend pods. While this incident has been resolved, we are not satisfied by this sequence of events, from both the risk management perspective and the detection / response perspective. We are investing in the following mechanisms to improve: * We have increased baseline server capacity and reverted the change to the feature flag that increased database load. We are also optimizing database connection usage to handle traffic spikes via better pooling and reuse. * Our monitoring around system load and client connection patterns will be made more comprehensive, including proactive alerts on key server-side load and performance metrics for request processing, so we can identify and react to issues before they impact service. * We are improving our client-server reconnection protocol during releases to include rate limiting and exponential backoff, and investigating further advancements so we can handle deployment rollovers more gracefully without service disruption. * Until the improvements above are in place, we will be avoiding deployments during peak traffic times, unless absolutely necessary to resolve major issues with service stability.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-0351e44027
Is Hex still down?
No, this incident was resolved 2025-08-20 18:26 UTC. For Hex's live state, see https://down.guide/hex.
How long did it last?
1h 26m, from 2025-08-20 17:00 UTC to 2025-08-20 18:26 UTC, by the times Hex posted.
What was affected?
Hex named Main site.
What caused it?
Hex hasn't published a root cause. If it does, it shows here.
Where does this come from?
Hex's official status page, https://status.hex.tech, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.