# Major systems outage

- Provider: [Apify](https://down.guide/apify)
- Level: confirmed
- Impact: Major outage
- lasted 1h 26m
- started 2025-02-22 22:12 UTC
- Web (apify.com), Datacenter, Dataset +9
- id: DWN-59679cb928
- Official page: https://stspg.io/mlqbfmm8klx1

## Updates from Apify
### 2025-02-22 22:12 UTC: investigating
We are investigating the issue.

### 2025-02-22 22:13 UTC: investigating
We are continuing to investigate this issue.

### 2025-02-22 23:05 UTC: identified
We've identified the cause of the issue and are taking steps to resolve it.

### 2025-02-22 23:25 UTC: monitoring
A fix has been implemented and we're monitoring the results. It might take some time to allocate all jobs, until all services have fully recovered.
We're sorry for the inconvenience. We will implement new measures based on this critical incident to prevent similar incidents in the future.

### 2025-02-22 23:33 UTC: monitoring
We are continuing to monitor for any further issues.

### 2025-02-22 23:38 UTC: resolved
This incident has been resolved.

### 2025-02-26 14:21 UTC: postmortem
Saturday outage post-mortem 
Date of incident: 2025-02-22
Impact: 
\* Complete downtime of the Apify platform from 22.00 UTC to 23.00 UTC.
\* No Actors could be started. 
\* Those already running were paused and continued running after the incident was resolved. 
\* There was no data loss, but the workloads were disrupted.
What happened:
\* Our user base is growing fast, and the number of new signups is increasing exponentially. The exponentially increased load revealed suboptimalities that had not manifested before.
\* One misconfigured query then caused a peak in memory due to a load of a suboptimal index, which caused the cluster to crash.
What we did:
\* The cluster size was increased to ensure enough memory until the issue was fixed. 
\* Then, the cluster and all the dependent systems were restarted.
Next steps:
\* We are already re-architecting our primary database clusters to optimize systems for future growth.
\* As part of this, we aim to decrease the size of all our clusters to ensure performant restarts in case of a problem.
\* We are currently optimizing indexing and querying among heavy-load components.
\* In addition, we identified missing metrics that would indicate similar problems ahead of time.
We sincerely apologize for the disruption and appreciate your patience. If you have any questions, please reach out to \[support@apify.com\](mailto:support@apify.com).
Sincerely, 
Apify engineering team

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).