# Ingestion and web UI outage

- Provider: [ServiceNow Cloud Observability](https://down.guide/lightstep)
- Level: confirmed
- Impact: Partial outage
- lasted 57m
- started 2022-07-21 20:05 UTC
- US - Trace Assembly, PagerDuty Events API, US - Service Directory +12
- us
- id: DWN-2235c726e2
- Official page: https://stspg.io/zdcb2q1tz85w

## Updates from ServiceNow Cloud Observability
### 2022-07-21 19:35 UTC: investigating
We are currently investigating this issue.

### 2022-07-21 19:35 UTC: investigating
We are continuing to investigate this issue.

### 2022-07-21 19:48 UTC: investigating
We are still actively investigating this issue, and will provide another update shortly.

### 2022-07-21 20:05 UTC: investigating
We have now identified the issue and are actively implementing mitigations.

### 2022-07-21 20:20 UTC: identified
We have mitigated issues with metric ingestion. Trace ingestion, assembly and alerting are still experiencing degraded performance for some customers.

### 2022-07-21 20:30 UTC: monitoring
We have resolved issues with trace ingestion, assembly and alerting and believe the issue has been resolved, but are actively monitoring.

### 2022-07-21 21:02 UTC: resolved
All components are operational.

### 2022-07-29 00:21 UTC: postmortem
### Summary
Lightstep UI and Data Ingest experienced an outage triggered by a planned database change on July 21st. This database change caused a service failure, which led to a cascading failure in dependent services. Action was taken to restore the service in a degraded mode, which recovered the Lightstep UI. A subsequent roll back of the database change restored all systems and services.
### Timeline
12:22 PM: A database change was run causing a service to become unavailable.
12:25 PM: Cascading failures impact Lightstep UI and ingestion. Ingested data loss begins.
12:35 PM: Incident declared in status page.
12:49 PM: Root cause identified, service brought back in a degraded mode, recovery begins.
13:01 PM: Several systems stabilize, Lightstep UI recovers.
13:17 PM: Database change rolled back. Remaining affected services begin recovering.
13:30 PM: Ingestion recovers. Incident resolved, status page updated to “mitigated and monitoring”.
14:02 PM: Status page updated to fully operational.
### Action Items
\* Adding automated checks on code changes with database changes to help prevent the original root cause
\* Enabling the directly affected service to continue to serve traffic in a degraded state, rather than failing entirely.
\* Making architectural changes to Lightstep’s Data Ingest path to prevent data loss in these situations by continuing to accept and buffer incoming traffic.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).