- Level
- confirmed
- Worst impact
- Degraded
- Components
- us-east
- Regions
- us
- Down Index
- 21of 100
What to do
Check whether you use Stream us-east in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 47m it lasted.
Anything that failed between 2020-01-28 16:12 UTC and 2020-01-28 16:58 UTC may need a replay.Subscribe to Stream's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=stream works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Stream 5
- monitoring
A recent released caused load increase on part of the chat infrastructure and caused degraded performance and timeout errors. Remediation is in progress.
provider
- monitoring
We are continuing to monitor for any further issues.
provider
- monitoring
We are continuing to monitor for any further issues.
provider
- resolved
This incident has been resolved.
provider
- postmortem
Between 4:05PM and 4:45PM UTC on January 28 2020 we had an API outage caused by performance degradation. The event was triggered by a new release to our Chat API servers; quickly after the new release was live, load on our database infrastructure increased and caused HTTP response times to spike and time-out in some cases. The event was detected by our latency and error monitoring. The team started working on the event by rolling back to the previous version at 4:20PM UTC. Unfortunately the rollback did not resolve the problem entirely. After another rollback attempt we realised there were still pending queries from the previous release running on our PostgreSQL database. We manually terminated all the pending tasks at 4:40PM UTC; after that the error rate dropped to 0% again. The outage affected 5% of HTTP requests at its peak \(4:20PM to 4:27PM UTC\).
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-3259497076
Is Stream still down?
No, this incident was resolved 2020-01-28 16:58 UTC. For Stream's live state, see https://down.guide/stream.
How long did it last?
47m, from 2020-01-28 16:12 UTC to 2020-01-28 16:58 UTC, by the times Stream posted.
What was affected?
Stream named us-east in us.
What caused it?
Stream hasn't published a root cause. If it does, it shows here.
Where does this come from?
Stream's official status page, https://status.getstream.io, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.