- Level
- confirmed
- Worst impact
- Partial outage
- Components
- US - Trace Assembly, US - Trace Statistics
- Regions
- us
- Down Index
- 35of 100
What to do
Check whether you use ServiceNow Cloud Observability US - Trace Assembly, US - Trace Statistics in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 4h 7m it lasted.
Anything that failed between 2022-02-04 20:28 UTC and 2022-02-05 00:35 UTC may need a replay.Subscribe to ServiceNow Cloud Observability's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=lightstep works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from ServiceNow Cloud Observability 4
- identified
The issue has been identified and a fix is being implemented.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- resolved
This incident has been resolved.
provider
- postmortem
## **Summary** A sharp increase in traffic overloaded one of our backend services responsible for communicating with satellites. The unavailability of the service led to a negative feedback loop as satellites attempted to reconnect to the failing service. During this time Lightstep experienced a partial outage of trace assembly and ingestion of statistics for streams. ## **Timeline \(12-hour Pacific Time\)** _~10:00a.m. Traffic to our SAAS sharply increases, overloading one of our backend services. 10:30a.m. The backend services restart, kicking off a cascading failure and starting the outage. 10:45a.m. Lightstep deploys more resources to an adjacent service which helps alleviate the symptoms of the outage and gets data flowing back into the system._ _11:20a.m. The rollout is complete and the cascading failure has resolved itself._ _12:07p.m. Lightstep marks the original incident as resolved as the symptoms have disappeared._ _12:24p.m. Lightstep increases resources for the root-cause service to reduce the likelihood of future outages. The change is deployed incorrectly and the service’s resources are decreased, which resumes the outage._ _12:28p.m. Lightstep updates the status page to reflect the new ongoing incident._ _12:51p.m. Lightstep identifies the error in the configuration, corrects it, and deploys the service with increased resources._ _12:58p.m. The service has finished rolling out and the outage is over._ **Action items** We have outlined 10 action items to avoid outages like this. Some examples include allocating more resources to the failing service, improving the autoscaling and alerting on the failing service, improving our autoscaling tools on adjacent services to be able to handle a failure in the root-cause service, and preventing OOMs of customer satellites when they cannot reach our SAAS. We’re also updating our incident response workflows to make it harder to incorrectly deploy bad configurations.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-0d6789053a
Is ServiceNow Cloud Observability still down?
No, this incident was resolved 2022-02-05 00:35 UTC. For ServiceNow Cloud Observability's live state, see https://down.guide/lightstep.
How long did it last?
4h 7m, from 2022-02-04 20:28 UTC to 2022-02-05 00:35 UTC, by the times ServiceNow Cloud Observability posted.
What was affected?
ServiceNow Cloud Observability named US - Trace Assembly, US - Trace Statistics in us.
What caused it?
ServiceNow Cloud Observability hasn't published a root cause. If it does, it shows here.
Where does this come from?
ServiceNow Cloud Observability's official status page, https://status.lightstep.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.