- Level
- confirmed
- Worst impact
- Partial outage
- Components
- Upload API
- Down Index
- 34of 100
What to do
Check whether you use Uploadcare Upload API. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 9h 29m it lasted.
Anything that failed between 2018-12-12 23:34 UTC and 2018-12-13 09:03 UTC may need a replay.Subscribe to Uploadcare's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=uploadcare works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Uploadcare 6
- investigating
We're experiencing issues with out Upload API.
provider
- investigating
We are continuing to investigate this issue.
provider
- identified
The issue has been identified and a fix is being implemented.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- resolved
This incident has been resolved.
provider
- postmortem
On December 12, we had a degradation of our Upload API. Most users were unable to upload files for 3 hours 11 minutes between Dec 12, 22:39 GMT and Dec 13, 01:50 GMT. ## What happened Requests to Upload API were either handled extremely slowly or \(most of them\) rejected by our web workers. Scaling up our Upload API fleet didn't help. ## What really happened Further investigation revealed that: — Slow requests were consuming all available web workers and without available workers requests were rejected by nginx. — Handled requests were slow due to constant database locks on one database table. — Database locks were caused by dramatical change of tracked usage statistics \(change of project settings by one of our largest customers\). We've spent most of the time during incident on investigation and figuring our what is happening. Actual DB load was average, and DB was wrongly dismissed as source of issues at first. Once we've the root cause, the fix was trivial and took minutes to implement and deploy. ## What we have done We turned off usage tracking for particular customer. ## What we will do — Refactor statistic tracking, so it does not affect our core service. — Add more specific monitors to our DB, so we could identify problems of similar nature much faster.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-0a599fdaa2
Is Uploadcare still down?
No, this incident was resolved 2018-12-13 09:03 UTC. For Uploadcare's live state, see https://down.guide/uploadcare.
How long did it last?
9h 29m, from 2018-12-12 23:34 UTC to 2018-12-13 09:03 UTC, by the times Uploadcare posted.
What was affected?
Uploadcare named Upload API.
What caused it?
Uploadcare hasn't published a root cause. If it does, it shows here.
Where does this come from?
Uploadcare's official status page, https://status.uploadcare.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.