Failing to ingest some span data

What to do

• Check whether you use ServiceNow Cloud Observability US - Trace Assembly in us. Check your stack

  Drop a package.json, lockfile, Terraform or a list of domains at down.guide to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 26m it lasted.

  Anything that failed between 2022-07-21 10:15 UTC and 2022-07-21 11:41 UTC may need a replay.

• Subscribe to ServiceNow Cloud Observability's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=lightstep works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from ServiceNow Cloud Observability 4

1. investigatingJul 21, 2022, 10:15 AM UTC

   We have identified the issue and are monitoring a fix.

   provider

2. monitoringJul 21, 2022, 10:49 AM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. resolvedJul 21, 2022, 11:41 AM UTC

   This incident has been resolved.

   provider

4. postmortemJul 29, 2022, 6:36 PM UTC

   Summary Increased span query load led to backpressure on span data writes. This backpressure led to resource exhaustion in one fleet of span ingestion pods, which caused a partial failure to ingest span data. The issue was resolved by scaling up the ingest pool. Timeline \(Pacific Time\) _2:45AM: Expensive queries lead to a spike in CPU utilization for a database pool. This spike leads to load shedding for writes, causing span ingestion errors._ _3:14AM: Scaled up the CPU for the affected pool._ _3:33AM: Scaled up the span ingestion pods._ _4:15AM: All systems are healthy again._ Action items We have compiled a list of action items, including prioritizing ingest over queries and improving retry handling, to more gracefully handle increased load.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.