# Elevated rendering errors - Provider: [imgix](https://down.guide/imgix) - Level: confirmed - Impact: Partial outage - lasted 2h 43m - started 2025-06-12 18:09 UTC - Rendering Infrastructure, Web Administration Tools, API Service - eu, us, us-central1 - id: DWN-910a146ec1 - Official page: https://stspg.io/qm9g6cjzq2nl ## Updates from imgix ### 2025-06-12 18:09 UTC: investigating We are investigating elevated render error rates for the service. Previously cached derivatives are not impacted. ### 2025-06-12 18:15 UTC: identified The issue has been identified and we are investigating a solution. ### 2025-06-12 18:53 UTC: identified The service is experiencing elevated error rates due to a major Google Cloud outage affecting services downstream. Previously cached derivatives are not impacted. We are investigating ways to mitigate this issue. ### 2025-06-12 20:26 UTC: monitoring The service is restored. We are monitoring the situation. ### 2025-06-12 20:38 UTC: identified The Rendering API has fully recovered. We are continuing to investigate Web Administration related issues (login and Management API) related to the incident. ### 2025-06-12 20:47 UTC: monitoring The Rendering API is fully recovered. Web Administration tooling (logins and the Management API) are recovering. We are monitoring the results. ### 2025-06-12 20:52 UTC: resolved The service is completely restored. ### 2025-06-20 19:03 UTC: postmortem # Incident Summary Between \*\*17:55 and 20:22 UTC\*\* on \*\*June 12, 2025\*\*, Imgix services experienced major service disruptions across several key interfaces: \* \*\*Dashboard and Asset Manager\*\*: These interfaces were inaccessible, preventing users from managing their assets or viewing account information. \* \*\*Management API\*\*: Requests to the Management API consistently returned errors, affecting workflows reliant on programmatic updates or asset administration. \* \*\*Rendering API:\*\* Approximately 8% of all Imgix service requests failed due to a high error rate \\(~80%\\) for \*\*uncached\*\* assets via the Rendering API. Requests in the EU saw a lower failure rate \\(~50%\\) and a faster recovery time \\(45 minutes\\) for these uncached requests. # What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in all regions simultaneously. You can read more about the \[Google Cloud outage here\](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1). # What happened \* \*\*17:55 UTC:\*\* Internal alerts triggered due to a spike in rendering errors and service timeouts. \* \*\*18:01 UTC:\*\* A short investigation uncovers several timeouts and increased error rates from Google Cloud. \* \*\*18:09 UTC:\*\* Our status page is updated. \* \*\*18:53 UTC:\*\* A major Google Cloud outage is confirmed, after which we update our status page. \* \*\*18:00–20:16 UTC:\*\* Mitigation efforts were hampered by the far-reaching effects of the outage, preventing us from redirecting traffic or applying configuration changes. \* \*\*Throughout:\*\* We confirm that cached images were not affected, though the downtime of several data sources prevents evaluating the full scope and effect of the outage. \* \*\*20:16 UTC:\*\* Google reported recovery in all regions except \`us-central1\`. This allowed us to verify \*\*significantly lower error rates for EU traffic\*\* \* \*\*20:47 UTC:\*\* Imgix systems achieved full recovery. \* \*\*20:52 UTC:\*\* The incident was officially resolved on our status page. \* \*\*21:23 UTC:\*\* Google confirms a full-service recovery at 21:23 UTC. # What went wrong \* Google Cloud experienced an outage that simultaneously affected nearly every service in every region worldwide, negating our multi-region redundancy for the image rendering service. \* 3rd party services \\(such as our CDN\\) were also affected by the outage, which removed some of our options for redirecting traffic across regions based on performance. \* The outage included the control panes that Google provides to its customers, which removed additional options for redirecting traffic and implementing mitigations. # What we will do to prevent this in the future \* Continuing our ongoing internal discussions and evaluations of a multi-cloud render stack to enable failover in the event of a provider-wide outages. \* Continue evaluating and improving tools to automatically and manually shift traffic as necessary at each layer of the stack. \* Review and enhance incident communication protocols, focusing on faster root cause disclosure and update frequency. --- down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).