Elevated rendering errors

What to do

• Check whether you use imgix Rendering Infrastructure. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 58m it lasted.

  Anything that failed between 2024-06-17 00:21 UTC and 2024-06-17 01:19 UTC may need a replay.

• Subscribe to imgix's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=imgix works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from imgix 6

1. investigatingJun 17, 2024, 12:21 AM UTC

   We are currently investigating elevated render error rates for uncached derivative images. We will update once when we obtain more information. Previously cached derivatives are not impacted.

   provider

2. identifiedJun 17, 2024, 12:50 AM UTC

   The issue has been identified and our engineering team is developing a fix.

   provider

3. identifiedJun 17, 2024, 12:51 AM UTC

   We are continuing to work on a fix for this issue.

   provider

4. monitoringJun 17, 2024, 1:06 AM UTC

   A fix has been implemented and error rates are returning to normal. We are continuing to monitor the service.

   provider

5. resolvedJun 17, 2024, 1:19 AM UTC

   This incident has been resolved.

   provider

6. postmortemJun 21, 2024, 11:54 PM UTC

   ## What happened? On June 17, 2024, at 00:00 UTC, imgix experienced an extreme spike in requests to our render stack. This unexpected surge caused a failure in our auto-scaling infrastructure, leading to an inability to manage all incoming traffic effectively. A fix was implemented at 00:38 UTC, and the issue was resolved by 01:06 UTC. ## How were customers impacted? Between 00:00 and 01:06 UTC, customers may have experienced failures when requesting new renders. However, previously cached assets served successfully during this time. ## What went wrong during the incident? The incident was triggered by a significant increase in requests, which our automated systems did not properly handle. Although the system started to auto-scale as expected, the unexpected surge caused issues with the health checks used for auto-scaling. The combination of extra traffic and health check failure led to an inability to render new images that required manual intervention to resolve. ## What will imgix do to prevent this in the future? To avoid similar incidents in the future, imgix is taking the following actions: 1. **Health Check Enhancement:** We have investigated and implemented updated health checks to support increased traffic volumes. 2. **Rate Limiting:** Further rate limits will be applied to manage traffic spikes and minimize their impact. 3. **Traffic Routing:** Traffic will be rerouted as necessary to distribute the load and reduce the risk of system overloads. 4. **Automated Alerts Improvement:** We will enhance our automated alert systems to respond more effectively to traffic surges and potential issues, including health check failures. By addressing these areas, we aim to further improve our system's resilience and ensure a smoother customer experience during periods of high demand.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.