Intermittent 5xx errors

What to do

• Check whether you use imgix Rendering Infrastructure. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 1h 15m it lasted.

  Anything that failed between 2023-05-01 14:20 UTC and 2023-05-01 15:35 UTC may need a replay.

• Subscribe to imgix's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=imgix works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from imgix 5

1. investigatingMay 1, 2023, 2:20 PM UTC

   We are currently investigating reports of intermittent 5xx errors causing some images to initially return a 5xx error.

   provider

2. identifiedMay 1, 2023, 2:47 PM UTC

   The issue has been identified, and a fix is being implemented.

   provider

3. monitoringMay 1, 2023, 3:08 PM UTC

   A fix has been implemented, and we are monitoring the results.

   provider

4. resolvedMay 1, 2023, 3:35 PM UTC

   This incident has been resolved.

   provider

5. postmortemMay 11, 2023, 8:37 PM UTC

   # What happened On May 1st, 2023, between the hours of 08:23 UTC and 15:08 UTC, imgix experienced intermittent errors affecting a small percentage of non-cached renders.  # How were customers impacted? During the affected period, a small percentage of requests to the Rendering API returned a `502` or `503` error for non-cached requests. Errors slowly and gradually increased, with <.5% of requests returning an error at the height of the incident. # What went wrong during the incident? Our upstream provider experienced communication issues between CDN POPs, causing intermittent `502`/`503` responses in a small percentage of requests to our Rendering API. The increase in errors was so minor that it did not meet our monitoring thresholds for triggering alerts. One of our engineers observed a slow increase in errors and alerted other team members to a potential issue with our service. After tracing the issue to our upstream provider, we pushed a patch to mitigate intermittent connectivity issues, resolving the incident. # What will imgix do to prevent this in the future? We have refined our alerting to better catch the slowly increasing error rates. We have also ensured that the root cause of this incident has been fixed by our upstream provider. We are also updating our traffic routing in the case that the upstream issue occurs again.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.