Degraded Performance of Phrase Orchestrator (EU & US) Workflow Engine Component between October 10, 10:10 AM CEST and October 10, 01:00 PM CEST

What to do

• Check whether you use Phrase Legacy Workflow Engine, Legacy Workflow Engine in eu, us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 42m it lasted.

  Anything that failed between 2025-10-10 10:52 UTC and 2025-10-10 13:34 UTC may need a replay.

• Subscribe to Phrase's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=phrase works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Phrase 4

1. investigatingOct 10, 2025, 10:52 AM UTC

   Phrase Orchestrator (EU & US) has been experiencing degraded performance. The engineering team is investigating the issue. We apologize for any inconvenience caused.

   provider

2. monitoringOct 10, 2025, 11:21 AM UTC

   Recovery is in progress across both EU and US regions, and the queues are currently being processed. Customers might still experience delayed workflow executions while the backlog is cleared. We are closely monitoring the situation to ensure full stability.

   provider

3. resolvedOct 10, 2025, 1:34 PM UTC

   This incident has been resolved.

   provider

4. postmortemOct 14, 2025, 11:50 AM UTC

   # **Root Cause Analysis** October 10, 2025 ### **Introduction** We would like to share more details about the events that occurred with Phrase Orchestrator \(EU & US\) between 10:05 AM CEST and 01:00 PM CEST on October 10, 2025 which led to delayed workflow executions for non-scheduled workflows and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 10:05 AM CEST: Our logging system showed repeated errors after introducing a change to Orchestrator’s system dependencies. 11:54 AM CEST: Through our monitoring system, Orchestrator engineers observed that workflows were not being processed by the Orchestrator workflow engine component. They began investigating and quickly identified the root cause - a recent system dependency change. 12:09 PM CEST: Our engineers prepared a rollback of the problematic change. 01:00 PM CEST: The rollback was successful. The workflow engine component began processing queued workflows again. 03:15 PM CEST: Orchestrator engineers confirmed the queue had cleared. ### **Root Cause** There was an update to a dependency which checks if the Workflow Engine component is under undue load. Due to a change in behavior of this dependency, the Workflow Builder component was unable to receive accurate load data from the Workflow Engine component. A, as a consequence, Orchestrator’s fail-safe mechanism activated to prevent data loss, causing no further workflows to execute in the workflow engine. ### **Actions to Prevent Recurrence** To prevent similar incidents in the future will improve our alerting around error logs. Additionally, we have identified further improvements for our alerting system that will notify us earlier in case of workflows not being processed. We will also speed up the process of rolling back releases in order to revert changes faster.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.