Quality degradation for a small portion of rendered images in the EU

What to do

• Check whether you use imgix in eu. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2024-12-13 17:57 UTC and 2024-12-10 03:00 UTC may need a replay.

• Subscribe to imgix's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=imgix works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from imgix 2

1. resolvedDec 10, 2024, 3:00 AM UTC

   A small percentage of image requests served by a single machine in the EU region experienced quality degradation between December 9th, 11:08 AM UTC - December 10th, 6:38 AM UTC

   provider

2. postmortemDec 13, 2024, 5:58 PM UTC

   # Timeframe December 9th, 11:08 AM UTC - December 10th, 6:38 AM UTC # Impact A small percentage of image requests served by a single machine in the EU region experienced quality degradation. # Root Cause A hardware failure on one of the rendering machines resulted in images with artifacts being delivered to users. # Contributing Factors An absence of automated testing for faulty renders across active machines caused assets to be cached undetected. The lapse between when the issue was detected and when the issue began culminated in an unknown number of renders that needed to be purged from the cache. Along with inadequate cache tooling for purging affected images, these factors caused the issue to fully resolve in a longer timeframe than expected. # Corrective Actions To prevent this issue from happening in the future, we will: * Investigate and implement automated image quality testing for active rendering machines * Replace and refresh machines on a more frequent basis to minimize impact of single machine failures * Upgrade cache tooling for targeted issue resolution

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.