Intermittent rendering errors

What to do

• Check whether you use imgix Rendering Infrastructure in us. Check your stack

  Drop a package.json, lockfile, Terraform or a list of domains at down.guide to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 59m it lasted.

  Anything that failed between 2023-10-23 22:34 UTC and 2023-10-23 23:32 UTC may need a replay.

• Subscribe to imgix's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=imgix works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from imgix 4

1. investigatingOct 23, 2023, 10:34 PM UTC

   We are investigating an issue affecting a small percentage of renders.

   provider

2. monitoringOct 23, 2023, 11:15 PM UTC

   A fix has been implemented and we are monitoring the results.

   provider

3. resolvedOct 23, 2023, 11:32 PM UTC

   This incident has been resolved.

   provider

4. postmortemNov 2, 2023, 6:52 PM UTC

   What happened? On October 23, 2023, between 21:43 UTC and 23:14 UTC, imgix experienced a partial outage affecting images served from the Rendering API. During this time, a small percentage \(<0.45% on average\) of non-cached requests returned a server error. A fix was implemented at 23:02 UTC, which allowed the service to recover by 23:14 UTC fully. How were customers impacted? Between 21:46 UTC and 23:14 UTC, requests to the Rendering API returned a server error, with 0.65% of all requests to our CDN returning an error at the height of the incident. Additionally, Sources returned an unknown status between 21:06 UTC to 21:09 UTC. During this period, customers reported being unable to create Sources. What went wrong during the incident? Our Rendering API experienced an unexpected interaction that caused a dramatic increase in server load. This caused error rates to increase as the network became overloaded slowly. The errors fluctuated between 0.07% to 0.65% until we resolved the issue. To restore the service, our engineers re-configured our network traffic to handle the unexpected Rendering behavior. During the incident, a separate issue \(unrelated to rendering\) impacted our Source data. This led to a delay in investigating the cause of the rendering errors. What will imgix do to prevent this in the future? We have taken the following steps to prevent this issue from recurring: * Fixed the misconfigured server interaction * We will put an alert system in place to notify us when traffic congestion happens from a misconfigured source interaction. We are in the process of implementing the following: * Conducting a review of our current tooling to increase our traffic and network configuration capabilities. * Reviewing our current configuration to limit the affected services should a similar incident happen.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.