# Failing to ingest some span data

- Provider: [ServiceNow Cloud Observability](https://down.guide/lightstep)
- Level: confirmed
- Impact: Partial outage
- lasted 1h 26m
- started 2022-07-21 10:15 UTC
- US - Trace Assembly
- us
- id: DWN-58c81763f8
- Official page: https://stspg.io/66899fzvzhw7

## Updates from ServiceNow Cloud Observability
### 2022-07-21 10:15 UTC: investigating
We have identified the issue and are monitoring a fix.

### 2022-07-21 10:49 UTC: monitoring
A fix has been implemented and we are monitoring the results.

### 2022-07-21 11:41 UTC: resolved
This incident has been resolved.

### 2022-07-29 18:36 UTC: postmortem
## Summary
Increased span query load led to backpressure on span data writes. This backpressure led to resource exhaustion in one fleet of span ingestion pods, which caused a partial failure to ingest span data. The issue was resolved by scaling up the ingest pool.
## Timeline \\(Pacific Time\\)
\_2:45AM: Expensive queries lead to a spike in CPU utilization for a database pool.  This spike leads to load shedding for writes, causing span ingestion errors.\_
\_3:14AM: Scaled up the CPU for the affected pool.\_
\_3:33AM: Scaled up the span ingestion pods.\_
\_4:15AM: All systems are healthy again.\_
## Action items
We have compiled a list of action items, including prioritizing ingest over queries and improving retry handling, to more gracefully handle increased load.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).