# REST API and Upload API outage

- Provider: [Uploadcare](https://down.guide/uploadcare)
- Level: confirmed
- Impact: Partial outage
- lasted 0m
- started 2024-07-08 09:31 UTC
- id: DWN-191170be6d
- Official page: https://stspg.io/6vw4b6v116fv

## Updates from Uploadcare
### 2024-07-08 09:31 UTC: resolved
2024-07-08 09:01:53 UTC
REST API and Upload API became unavailable due to system misconfiguration. 
2024-07-08 09:33:12 UTC
We've identified the source of problems, deployed fixes. Service recovered. We are monitoring the situation.

### 2024-07-16 14:45 UTC: postmortem
## REST and Upload API service degradation \\(incident #wvjpwt1qtpkn\\)
\*\*Date\*\*: 2024-07-08
\*\*Authors\*\*: Arsenii Baranov
\*\*Status\*\*: Complete
\*\*Summary\*\*: From 09:01:53 to 09:33:12 UTC we've experienced higher latencies and in the end a complete outage of Upload and REST API.
\*\*Root Causes\*\*: PostgreSQL performance degradation due to human-factor.
\*\*Trigger\*\*: Misconfiguration.
\*\*Resolution\*\*: Changed our DBA-related processes, fixed monitoring related issue.
\*\*Detection\*\*: Our Customer Success team detected the issue and escalated to the Engineering team.
\*\*Action Items\*\*:
\| Action Item \| Type \| Status \|
\| --- \| --- \| --- \|
\| Fix monitoring misconfiguration \| mitigate \| \*\*DONE\*\* \|
\| Improve DBA maintenance approach \| prevent \| \*\*DONE\*\* \|
## Lessons Learned
\*\*What went well\*\*
\* Due to distributed nature of Uploadcare, this incident has no effect on most of our services. This degradation didn’t affect storage, processing and serving files that were already stored by Uploadcare CDN.
\* Our incident mitigation strategy was right and worked immediately.
\*\*What went wrong\*\*
\* This incident was detected in non-automatic way due to alert misconfiguration.
\* We failed to process API request during the incident.
## Timeline
2024-07-08 \_\\(all times UTC\\)\_
\* 08:00 Database maintenance started
\* 09:01 \*\*SERVICE DEGRADATION BEGINS\*\*
\* 09:23 Our customer success team escalates issue to Infrastructure team
\* 09:29 Issue localised and fixed
\* 09:33 \*\*SERVICE DEGRADATION ENDS\*\*

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).