Platform instability

What to do

• Check whether you use Hex Main site. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 51m it lasted.

  Anything that failed between 2024-11-12 16:18 UTC and 2024-11-12 19:08 UTC may need a replay.

• Subscribe to Hex's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=hex works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Hex 5

1. investigatingNov 12, 2024, 4:18 PM UTC

   We are currently investigating this issue.

   provider

2. investigatingNov 12, 2024, 4:18 PM UTC

   We are continuing to investigate this issue.

   provider

3. investigatingNov 12, 2024, 5:17 PM UTC

   We are continuing to investigate this issue.

   provider

4. monitoringNov 12, 2024, 5:38 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

5. resolvedNov 12, 2024, 7:08 PM UTC

   On November 12, 2024 from 8:06 AM to 10:21 PM PT, Hex customers experienced platform instability, including issues with kernel execution and site access. This disruption stemmed from an unintended effect of a recent database migration, which ultimately caused a backlog and impacted overall platform performance. Our engineering team promptly identified the root cause and took corrective actions by resetting kernels to alleviate database congestion. Once the database stabilized, we rolled back the migration change, fully restoring service by 10:21 PM PT. To prevent similar issues in the future, we are enhancing our automated tests to better validate database migrations before deployment, ensuring they don’t inadvertently impact platform reliability. Thank you for your patience and understanding.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.