Performance Degradation

What to do

• Check whether you use Upstash N. Virginia, USA (us-east-1) in us, us-east-1. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 4h 45m it lasted.

  Anything that failed between 2025-06-10 07:23 UTC and 2025-06-10 12:08 UTC may need a replay.

• Subscribe to Upstash's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=upstash works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Upstash 4

1. investigatingJun 10, 2025, 7:23 AM UTC

   We are currently investigating this issue.

   provider

2. monitoringJun 10, 2025, 7:45 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. resolvedJun 10, 2025, 12:08 PM UTC

   This incident has been resolved.

   provider

4. postmortemJun 11, 2025, 11:37 AM UTC

   A routine system maintenance operation at the OS level led to the application of system updates across multiple EC2 instances in our clusters in several AWS regions. These updates included changes to networking components, which inadvertently triggered restarts. As a result, several EC2 nodes failed health checks and temporarily dropped out of the cluster, disrupting high availability and causing partial connectivity issues for some clients and operations. We have since reproduced the issue in a controlled environment and verified the root cause. To prevent a recurrence, we are updating our node maintenance strategy to ensure greater control over the timing and impact of system-level changes and excluding networking components from automated upgrades.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.