REST API and Upload API outage

What to do

• Check whether your product depends on Uploadcare. Check your stack

  Drop a package.json, lockfile, Terraform or a list of domains at down.guide to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2024-07-08 09:31 UTC and 2024-07-08 09:31 UTC may need a replay.

• Subscribe to Uploadcare's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=uploadcare works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Root cause

Root Causes: PostgreSQL performance degradation due to human-factor.

Updates from Uploadcare 2

1. resolvedJul 8, 2024, 9:31 AM UTC

   2024-07-08 09:01:53 UTC REST API and Upload API became unavailable due to system misconfiguration. 2024-07-08 09:33:12 UTC We've identified the source of problems, deployed fixes. Service recovered. We are monitoring the situation.

   provider

2. postmortemJul 16, 2024, 2:45 PM UTC

   REST and Upload API service degradation \(incident #wvjpwt1qtpkn\) Date: 2024-07-08 Authors: Arsenii Baranov Status: Complete Summary: From 09:01:53 to 09:33:12 UTC we've experienced higher latencies and in the end a complete outage of Upload and REST API. Root Causes: PostgreSQL performance degradation due to human-factor. Trigger: Misconfiguration. Resolution: Changed our DBA-related processes, fixed monitoring related issue. Detection: Our Customer Success team detected the issue and escalated to the Engineering team. Action Items: | Action Item | Type | Status | | --- | --- | --- | | Fix monitoring misconfiguration | mitigate | DONE | | Improve DBA maintenance approach | prevent | DONE | Lessons Learned What went well * Due to distributed nature of Uploadcare, this incident has no effect on most of our services. This degradation didn’t affect storage, processing and serving files that were already stored by Uploadcare CDN. * Our incident mitigation strategy was right and worked immediately. What went wrong * This incident was detected in non-automatic way due to alert misconfiguration. * We failed to process API request during the incident. Timeline 2024-07-08 _\(all times UTC\)_ * 08:00 Database maintenance started * 09:01 SERVICE DEGRADATION BEGINS * 09:23 Our customer success team escalates issue to Infrastructure team * 09:29 Issue localised and fixed * 09:33 SERVICE DEGRADATION ENDS

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.