- Level
- confirmed
- Worst impact
- Degraded
- Components
- Rendering Infrastructure
- Regions
- us
- Down Index
- 21of 100
What to do
Check whether you use imgix Rendering Infrastructure in us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 59m it lasted.
Anything that failed between 2023-10-23 22:34 UTC and 2023-10-23 23:32 UTC may need a replay.Subscribe to imgix's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=imgix works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from imgix 4
- investigating
We are investigating an issue affecting a small percentage of renders.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- resolved
This incident has been resolved.
provider
- postmortem
# What happened? On October 23, 2023, between 21:43 UTC and 23:14 UTC, imgix experienced a partial outage affecting images served from the Rendering API. During this time, a small percentage \(<0.45% on average\) of non-cached requests returned a server error. A fix was implemented at 23:02 UTC, which allowed the service to recover by 23:14 UTC fully. # How were customers impacted? Between 21:46 UTC and 23:14 UTC, requests to the Rendering API returned a server error, with 0.65% of all requests to our CDN returning an error at the height of the incident. Additionally, Sources returned an unknown status between 21:06 UTC to 21:09 UTC. During this period, customers reported being unable to create Sources. # What went wrong during the incident? Our Rendering API experienced an unexpected interaction that caused a dramatic increase in server load. This caused error rates to increase as the network became overloaded slowly. The errors fluctuated between 0.07% to 0.65% until we resolved the issue. To restore the service, our engineers re-configured our network traffic to handle the unexpected Rendering behavior. During the incident, a separate issue \(unrelated to rendering\) impacted our Source data. This led to a delay in investigating the cause of the rendering errors. # What will imgix do to prevent this in the future? We have taken the following steps to prevent this issue from recurring: * Fixed the misconfigured server interaction * We will put an alert system in place to notify us when traffic congestion happens from a misconfigured source interaction. We are in the process of implementing the following: * Conducting a review of our current tooling to increase our traffic and network configuration capabilities. * Reviewing our current configuration to limit the affected services should a similar incident happen.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-0725f9aa9e
Is imgix still down?
No, this incident was resolved 2023-10-23 23:32 UTC. For imgix's live state, see https://down.guide/imgix.
How long did it last?
59m, from 2023-10-23 22:34 UTC to 2023-10-23 23:32 UTC, by the times imgix posted.
What was affected?
imgix named Rendering Infrastructure in us.
What caused it?
imgix hasn't published a root cause. If it does, it shows here.
Where does this come from?
imgix's official status page, https://status.imgix.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.