- Level
- confirmed
- Worst impact
- Partial outage
- Components
- Global Web Application, Global API Success, Global System Webhooks, Global Partner Webhooks, Global Campaign Sends
- Down Index
- 33of 100
What to do
Check whether you use Iterable Global Web Application, Global API Success, Global System Webhooks, Global Partner Webhooks and 1 more. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 22h 52m it lasted.
Anything that failed between 2024-11-24 18:16 UTC and 2024-11-25 17:09 UTC may need a replay.Subscribe to Iterable's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=iterable works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Iterable 19
- investigating
Beginning at 5 AM PST we were alerted to a spike in API errors and Web App errors. Our engineering team is investigating. Next update will be at 7:30 AM PST or sooner. If you have questions please reach out to support@iterable.com
provider
- investigating
Iterable engineering has put in mitigations and is seeing recovery in the web application and API endpoints. We are continuing to work on remediating delays to sends and journey processing. Next update by 8:30 am PT
provider
- investigating
We're seeing continued improvements in the web application and API performance. We are still working to resolve delays in sends and webhooks. Next update by 9:00 AM PT or sooner
provider
- investigating
We have put in further mitigations to restore send speeds and are working through the send backlog. We are continuing to work on recovering webhooks. Next update by 9:30 AM PT or sooner
provider
- investigating
The Engineering team is investigating web and API errors. Next update by 9:45 AM PT or sooner.
provider
- investigating
The Engineering team is implementing mitigations to address the intermittent web and API errors as well as send and webhook delays. Next update by 10:15 AM PT or sooner.
provider
- investigating
The Engineering team is continuing to investigate and implement mitigations to address DB issues causing web and API errors. Sends, journey processing, and webhooks will continue to see delays. Customers may also experience issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 10:45 AM PT or sooner.
provider
- investigating
The Engineering team is continuing to investigate and implement mitigations to address DB issues causing web and API errors. Sends, journey processing, and webhooks will continue to see delays. Customers may also experience issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 11:15 AM PT or sooner.
provider
- investigating
The Engineering team is continuing to investigate and implement mitigations to address DB issues. Web errors and journey latency are going down and send rate is going up. Customers may still experience degradation and issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 11:45 AM PT or sooner.
provider
- identified
The Engineering team is continuing to address DB issues. We are slowly bringing up the scaled down instances of various services. Web errors and journey latency are going down and send rate is going up. Customers may still experience issue with web app failing to fully load and an inability to switch projects. Next update by 12:15 PM PT or sooner.
provider
- identified
The issue is now identified. The engineering team is slowly bringing up the scaled down instances of various services. Customers should begin seeing improvements in access to the web app, and message send rates are beginning to recover. All services except webhooks are scaled up now. We should continue to see improvements across the app as we catch up on the messaging backlog. Next update by 12:45 PM PT or sooner.
provider
- identified
The issue is identified. The engineering team continues to scale up instances of various services. Customers should begin seeing improvements in access to the web app, and message send rates are beginning to recover. We should continue to see improvements across the app as we catch up on the messaging backlog. Next update by 1:15 PM PT or sooner.
provider
- identified
The issue is identified. The engineering team continues to scale up instances of various services. Customers should continue seeing improvements in access to the web app, and message send rates continue to recover. We continue to see improvements across the app as we catch up on the messaging backlog. Next update by 2:15PM PT or sooner.
provider
- monitoring
The engineering team has completed the instances scale up of various services. Access to the web app, and message send rates are back to normal processing. We continue to see improvements across the app as we catch up on the messaging backlog. Most systems are caught up with processing. Next update by 3:15PM PT or sooner.
provider
- monitoring
Engineering team has confirmed that we continue to process at normal rates. Our processing of messaging backlog has caught up. Engineering is continuing to monitor our systems, if you are still seeing any impact please reach out to support@iterable.com
provider
- monitoring
We are continuing to monitor for any further issues.
provider
- monitoring
The Engineering team continues to monitor system performance. There have been no performance degradations. Next update by 12 pm PT or sooner. If you are still seeing any impact please reach out to support@iterable.com
provider
- monitoring
This incident has been resolved.
provider
- resolved
This incident has been resolved.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-af440e9313
Is Iterable still down?
No, this incident was resolved 2024-11-25 17:09 UTC. For Iterable's live state, see https://down.guide/iterable.
How long did it last?
22h 52m, from 2024-11-24 18:16 UTC to 2024-11-25 17:09 UTC, by the times Iterable posted.
What was affected?
Iterable named Global Web Application, Global API Success, Global System Webhooks, Global Partner Webhooks and 1 more.
What caused it?
Iterable hasn't published a root cause. If it does, it shows here.
Where does this come from?
Iterable's official status page, https://status.iterable.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.