- Level
- confirmed
- Worst impact
- Partial outage
- Components
- US - Trace Assembly, PagerDuty Events API, US - Service Directory, US - Unified Alerting, US - Dashboards, US - Streams Alerting, US - Trace Statistics, Slack Apps/Integrations, US - Notebooks, US - Metrics, US - Observability - API, Slack Connections, US - Change Intelligence, US - Explorer, Slack Messaging
- Regions
- us
- Down Index
- 36of 100
What to do
Check whether you use ServiceNow Cloud Observability US - Trace Assembly, PagerDuty Events API, US - Service Directory, US - Unified Alerting and 11 more in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 57m it lasted.
Anything that failed between 2022-07-21 20:05 UTC and 2022-07-21 21:02 UTC may need a replay.Subscribe to ServiceNow Cloud Observability's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=lightstep works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from ServiceNow Cloud Observability 8
- investigating
We are currently investigating this issue.
provider
- investigating
We are continuing to investigate this issue.
provider
- investigating
We are still actively investigating this issue, and will provide another update shortly.
provider
- investigating
We have now identified the issue and are actively implementing mitigations.
provider
- identified
We have mitigated issues with metric ingestion. Trace ingestion, assembly and alerting are still experiencing degraded performance for some customers.
provider
- monitoring
We have resolved issues with trace ingestion, assembly and alerting and believe the issue has been resolved, but are actively monitoring.
provider
- resolved
All components are operational.
provider
- postmortem
### Summary Lightstep UI and Data Ingest experienced an outage triggered by a planned database change on July 21st. This database change caused a service failure, which led to a cascading failure in dependent services. Action was taken to restore the service in a degraded mode, which recovered the Lightstep UI. A subsequent roll back of the database change restored all systems and services. ### Timeline 12:22 PM: A database change was run causing a service to become unavailable. 12:25 PM: Cascading failures impact Lightstep UI and ingestion. Ingested data loss begins. 12:35 PM: Incident declared in status page. 12:49 PM: Root cause identified, service brought back in a degraded mode, recovery begins. 13:01 PM: Several systems stabilize, Lightstep UI recovers. 13:17 PM: Database change rolled back. Remaining affected services begin recovering. 13:30 PM: Ingestion recovers. Incident resolved, status page updated to “mitigated and monitoring”. 14:02 PM: Status page updated to fully operational. ### Action Items * Adding automated checks on code changes with database changes to help prevent the original root cause * Enabling the directly affected service to continue to serve traffic in a degraded state, rather than failing entirely. * Making architectural changes to Lightstep’s Data Ingest path to prevent data loss in these situations by continuing to accept and buffer incoming traffic.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-2235c726e2
Is ServiceNow Cloud Observability still down?
No, this incident was resolved 2022-07-21 21:02 UTC. For ServiceNow Cloud Observability's live state, see https://down.guide/lightstep.
How long did it last?
57m, from 2022-07-21 20:05 UTC to 2022-07-21 21:02 UTC, by the times ServiceNow Cloud Observability posted.
What was affected?
ServiceNow Cloud Observability named US - Trace Assembly, PagerDuty Events API, US - Service Directory, US - Unified Alerting and 11 more in us.
What caused it?
ServiceNow Cloud Observability hasn't published a root cause. If it does, it shows here.
Where does this come from?
ServiceNow Cloud Observability's official status page, https://status.lightstep.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.