Elevated API Errors on us-east

What to do

• Check whether you use Stream us-east in us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 5h 23m it lasted.

  Anything that failed between 2021-12-08 07:20 UTC and 2021-12-08 12:43 UTC may need a replay.

• Subscribe to Stream's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=stream works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Stream 5

1. investigatingDec 8, 2021, 7:20 AM UTC

   We're experiencing an elevated level of API errors and are currently looking into the issue. This issue affects one shard only in our us-east region.

   provider

2. identifiedDec 8, 2021, 8:55 AM UTC

   The issue has been identified and a fix is being implemented. A temporary remediation has been put in place to mitigate the ongoing issue.

   provider

3. monitoringDec 8, 2021, 9:02 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

4. monitoringDec 8, 2021, 11:04 AM UTC

   The issue of this morning propagated to an additional component of our infrastructure intended to dispatch messages to the end users via websocket protocol. Our team tried to mitigate the issue and the problem seems to be resolved now. We are still monitoring the situation closely.

   provider

5. resolvedDec 8, 2021, 12:43 PM UTC

   The incident has been resolved. A post-mortem will follow.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.