Skip to main content
Iterable

Event-Triggered Journeys Delays Ingesting New Users

Partial outage Lasted 3h 56m · started · Iterable's page

Level
confirmed
Worst impact
Partial outage
Components
Global Web Application
Down Index
30of 100

What to do

  • Check whether you use Iterable Global Web Application. Check your stack

    Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.
  • Confirm your retries, queues and scheduled jobs caught up after the 3h 56m it lasted.

    Anything that failed between 2024-12-31 14:29 UTC and 2024-12-31 18:25 UTC may need a replay.
  • Subscribe to Iterable's feed to hear about the next one first. RSS feed

    https://down.guide/feed?provider=iterable works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Iterable 3

  1. identified

    Summary: Event-Triggered Journeys Delays Ingesting New Users. The issue originated from errors in the workflow-entrance-trigger pods, causing a significant backlog in processing. There is no impact to Scheduled Journeys and API Triggered Journeys . Actions Taken The workflow-entrance-trigger service was updated to the latest version, and additional pods were scaled up to process the backlog faster. The deployment resolved the issue, and error rates dropped significantly. Current Status The errors we were experiencing have been fixed since 5AM PST, now we're just monitoring the backlog as it drains. For 99% of clients, the backlog has drained completely, there are a few stragglers with small backlogs Next Steps Engineers will continue monitoring error rates and ensure the backlog clears entirely. Follow-up tasks include setting up error rate monitoring and addressing journey-specific issues to prevent recurrence.

    provider

  2. identified

    We are continuing to work on a fix for this issue.

    provider

  3. resolved

    This issue is now resolved and the backlog is completely drained. Event trigger journey's as of 7:25 AM PST are back to normal and processing as expected. If you still have any further questions please reach out to support@iterable.com.

    provider

Reports count in aggregate; nothing about you is shown.

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.

Questions about DWN-1464e01bcd

Is Iterable still down?

No, this incident was resolved 2024-12-31 18:25 UTC. For Iterable's live state, see https://down.guide/iterable.

How long did it last?

3h 56m, from 2024-12-31 14:29 UTC to 2024-12-31 18:25 UTC, by the times Iterable posted.

What was affected?

Iterable named Global Web Application.

What caused it?

Iterable hasn't published a root cause. If it does, it shows here.

Where does this come from?

Iterable's official status page, https://status.iterable.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.