- Level
- confirmed
- Worst impact
- Major outage
- Components
- US - Service Directory, US - Dashboards, US - Change Intelligence, US - Explorer
- Regions
- us
- Down Index
- 46of 100
What to do
Check whether you use ServiceNow Cloud Observability US - Service Directory, US - Dashboards, US - Change Intelligence, US - Explorer in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 3h 24m it lasted.
Anything that failed between 2022-01-20 17:58 UTC and 2022-01-20 21:21 UTC may need a replay.Subscribe to ServiceNow Cloud Observability's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=lightstep works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from ServiceNow Cloud Observability 6
- investigating
We are currently investigating this issue.
provider
- investigating
All pages are operational except for the streams page. We are continuing to investigate.
provider
- monitoring
A fix for the remaining unavailability on the streams page is rolling out, and is expected to complete by 1pm PST.
provider
- monitoring
We are continuing to monitor for any further issues.
provider
- resolved
This incident has been resolved.
provider
- postmortem
## **Summary** Web UI inaccessible for 48 minutes \(9:49am - 10:37am\), and streams page inaccessible for an additional 48 minutes \(until 11:25am\). ## **Timeline \(12-hour Pacific Time\)** 09:49 AM PT: 100% of API requests for the webapp begin failing 09:52 AM PT: Database hits 100% resource utilization 10:31 AM PT: Divert all traffic from webapp to bring up database 10:37 AM PT: Allow all traffic except from the operation/get endpoint so the webapp is back up 11:25 AM PT: Allow operation/get traffic and incident is resolved ## **Root Cause** Database errors led to transient unavailability. Aggressive retries saturated the database, leading to a negative feedback loop. ## **Action items** We have updated our database configuration to limit the blast radius of failing or slow database calls. We have also implemented additional rate limiting to avoid the negative feedback loop where many concurrent requests lead to failures, leading to more requests.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-c972089f00
Is ServiceNow Cloud Observability still down?
No, this incident was resolved 2022-01-20 21:21 UTC. For ServiceNow Cloud Observability's live state, see https://down.guide/lightstep.
How long did it last?
3h 24m, from 2022-01-20 17:58 UTC to 2022-01-20 21:21 UTC, by the times ServiceNow Cloud Observability posted.
What was affected?
ServiceNow Cloud Observability named US - Service Directory, US - Dashboards, US - Change Intelligence, US - Explorer in us.
What caused it?
ServiceNow Cloud Observability hasn't published a root cause. If it does, it shows here.
Where does this come from?
ServiceNow Cloud Observability's official status page, https://status.lightstep.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.