Performance degradation on QStash

What to do

• Check whether you use Upstash EU-CENTRAL-1 in eu, eu-central-1. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 34m it lasted.

  Anything that failed between 2025-03-12 09:43 UTC and 2025-03-12 10:16 UTC may need a replay.

• Subscribe to Upstash's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=upstash works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Upstash 4

1. investigatingMar 12, 2025, 9:43 AM UTC

   We are currently investigating this issue.

   provider

2. monitoringMar 12, 2025, 10:08 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. resolvedMar 12, 2025, 10:16 AM UTC

   This incident has been resolved.

   provider

4. postmortemMar 12, 2025, 12:26 PM UTC

   On 09:43 UTC, QStash experienced degraded service when a high number of requests to a specific domain were throttled, resulting in timeouts during an unexpected phase of the TCP connection establishment. These requests and resulting retries triggered excessive consumption of network resources and negatively impacted all users. We have added more resources to QStash as a quick remediation and as for the resolution, we have improved the timeout mechanism to detect and fail-faster in such cases.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.