- Level
- confirmed
- Worst impact
- Partial outage
- Components
- Coveralls.io Web, Coveralls.io API
- Down Index
- 38of 100
What to do
Check whether you use Coveralls Coveralls.io Web, Coveralls.io API. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 18h 57m it lasted.
Anything that failed between 2025-06-02 15:55 UTC and 2025-06-03 10:52 UTC may need a replay.Subscribe to Coveralls's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=coveralls works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Coveralls 16
- investigating
We are currently investigating this issue.
provider
- identified
The issue has been identified and a fix is being implemented.
provider
- identified
We need to pause processing momentarily to clear a backlog of DB connections. We cutover to a new database version this weekend and even after months of planning and preventative steps, during periods of elevated usage after such a change it's still common for planner regressions to occur. We will identify the offending SQL statements, fix their planner issues, and restart work as soon as possible. Thanks for your pateince as we work though this as quickly as possible.
provider
- identified
We are continuing to work on a fix for this issue.
provider
- identified
We are continuing to work on a fix for this issue.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- monitoring
We’re currently experiencing an outage due to unexpected query planner behavior following our recent upgrade to PostgreSQL 16. Despite extensive preparation and testing, one of our core background queries began performing full table scans under the new version, causing a rapid increase in load and job backlog. What we're doing: - We’ve paused background job processing to stabilize the system. - We tried all "quick fixes" like adjustments to DB params that affect planner choices—all to no effect. - We're now actively deploying a targeted database index to resolve the performance issue. - We’ve identified a longer-term fix that will make the query safer and more efficient on the new version of PostgreSQL. Why this happened: PostgreSQL 16 introduced changes to how certain types of queries are planned. A query that performed well in PostgreSQL 12 unexpectedly triggered a much more expensive plan in 16. We're correcting for that now. Estimated recovery: Background job processing is expected to resume within 20–40 minutes, with full service restoration shortly thereafter. We’ll continue to post updates here as we make progress. Thanks for your patience — we’re on it.
provider
- monitoring
We have completed implementation of our fix. We are cautiously resuming background processing and will continue monitoring closely. If you notice any delays in build processing, rest assured they will be resolved shortly. Thank you for your patience — more updates will follow as we return to full capacity.
provider
- monitoring
All systems operational. We are carefully scaling resources and monitoring database performance to ensure stable recovery. Some delays in build and coverage report processing may still be observed as we restore full capacity. Thank you for your continued patience — we’ll share further updates as recovery progresses.
provider
- investigating
While monitoring we have discovered some additional planner anomalies that are slowing down queries associated with our various calculation jobs. We are investigating those again and working to identify and implement a fix. We will continue posting updates here.
provider
- identified
The issue has been identified and a fix is being implemented.
provider
- identified
We are continuing to work on a fix for this issue.
provider
- monitoring
A partial fix has been implemented and we are monitoring the results.
provider
- monitoring
We are continuing to monitor for any further issues.
provider
- monitoring
We have implemented another fix and are monitoring the results.
provider
- resolved
This incident has been resolved, but we will continue monitoring closely. All systems are operational, but we will leave systems category at Degraded Performance until we have fully cleared a backlog of background processing jobs.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-50e986973f
Is Coveralls still down?
No, this incident was resolved 2025-06-03 10:52 UTC. For Coveralls's live state, see https://down.guide/coveralls.
How long did it last?
18h 57m, from 2025-06-02 15:55 UTC to 2025-06-03 10:52 UTC, by the times Coveralls posted.
What was affected?
Coveralls named Coveralls.io Web, Coveralls.io API.
What caused it?
Coveralls hasn't published a root cause. If it does, it shows here.
Where does this come from?
Coveralls's official status page, https://status.coveralls.io, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.