- Level
- confirmed
- Worst impact
- Degraded
- Components
- Data Processing, Message Sending
- Down Index
- 24of 100
What to do
Check whether you use Customer.io Data Processing, Message Sending. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 2h 6m it lasted.
Anything that failed between 2025-12-08 18:12 UTC and 2025-12-08 20:19 UTC may need a replay.Subscribe to Customer.io's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=customerio works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Customer.io 5
- investigating
We identified a recent change that might cause data delays. We are investigating.
provider
- identified
The issue has been identified and a fix is being implemented.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- resolved
This incident has been resolved.
provider
- postmortem
`Incident Summary ` `On December 8th, 2025, beginning at 17:38 UTC, some customers experienced delays in data processing and message delivery. Normal functionality was fully restored at 18:19 UTC, for a total duration of 41 minutes. No data was lost. ` `A failure during the startup of an internal processing service prevented it from becoming fully operational, leading to reduced throughput and increased retry activity in upstream components. ` `Root Cause ` `During startup, one of our processing services loads information about message queues before beginning normal operation. An unexpected queue state left over from a previous configuration caused the service to encounter an error during this process, leading it to restart repeatedly without successfully completing initialization. ` `Because this service is responsible for handing off work to downstream processors, its unavailability resulted in a drop in throughput and a rise in retry traffic. The elevated retries added load to our underlying data layer and contributed to further delays. ` `Resolution and Recovery ` `Engineers identified the failing service, corrected the underlying queue state, and restored the service to full operation. Once stabilized, normal processing resumed and retry volumes returned to expected levels. The system was monitored to confirm full recovery. ` `Corrective and Preventative Measures ` `To prevent recurrence, the team is improving validation during service startup to better handle unexpected queue conditions, refining deployment procedures to detect stalled services sooner, and enhancing monitoring for repeated restart patterns. These improvements are being incorporated into ongoing reliability work. ` `We apologize for any disruption this caused.`
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-ced605e36c
Is Customer.io still down?
No, this incident was resolved 2025-12-08 20:19 UTC. For Customer.io's live state, see https://down.guide/customerio.
How long did it last?
2h 6m, from 2025-12-08 18:12 UTC to 2025-12-08 20:19 UTC, by the times Customer.io posted.
What was affected?
Customer.io named Data Processing, Message Sending.
What caused it?
Customer.io hasn't published a root cause. If it does, it shows here.
Where does this come from?
Customer.io's official status page, https://status.customerio.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.