Degraded Performance of Phrase Orchestrator (EU) Next-Gen Workflow Engine between July 28, 05:15 AM CEST and July 28, 09:56 AM CEST

What to do

• Check whether you use Phrase Next-Gen Workflow Engine in eu. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 4h 30m it lasted.

  Anything that failed between 2026-07-28 07:19 UTC and 2026-07-28 11:49 UTC may need a replay.

• Subscribe to Phrase's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=phrase works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Phrase 6

1. investigatingJul 28, 2026, 7:19 AM UTC

   The engineering team identified an issue with Orchestrator where the new Workflow Engine is currently not executing workflows. The problem is under investigation.

   provider

2. investigatingJul 28, 2026, 7:53 AM UTC

   We are continuing to investigate this issue.

   provider

3. identifiedJul 28, 2026, 8:30 AM UTC

   The issue has been identified and a fix is being implemented.

   provider

4. monitoringJul 28, 2026, 8:35 AM UTC

   A fix has been implemented and we are monitoring the results. The workflow engine is processing the queue of pending executions.

   provider

5. resolvedJul 28, 2026, 11:49 AM UTC

   All executions have been processed. This incident has been resolved.

   provider

6. postmortemJul 30, 2026, 7:09 AM UTC

   ## Introduction We would like to share details about an incident that affected Phrase Orchestrator on July 27–28, 2026. During this period, workflow executions in the Next-Gen Workflow Engine were unable to progress and remained stuck in an "executing" state. No data was lost during the incident. This post-mortem explains what happened, when it was resolved, and the steps we have taken to prevent a recurrence. ## Timeline * **27 July 2026 at 18:55 CEST** – The workflow engine began producing errors as database query performance degraded. Workflow executions stalled and stopped progressing. * **27 July 2026 at 20:54 CEST** – The first customer report of executions stuck in "executing" was received. * **27 July 2026 at 22:39 CEST** – The incident was formally declared. * **28 July 2026 at 02:18 CEST** – A service restart provided temporary relief; workflow executions resumed. * **28 July 2026 at 05:15 CEST** – The issue recurred as the underlying database performance problem persisted. * **28 July 2026 at 09:56 CEST** – The root cause was identified and addressed. No executions were lost; however, due to partial service restarts, some actions within executions were retried, which may have caused a small number of executions to fail that otherwise would have succeeded. * **28 July 2026 at 13:46 CEST** – The full backlog of stalled executions was confirmed as cleared. The system was declared stable. * **28 July 2026 at 13:48 CEST** – Incident resolved. ## Root Cause The incident was caused by progressive bloat in database indexes used by the workflow job scheduling system. The performance of these particular indexes gradually degraded over time as they accumulated dead index entries from prior writes and updates. The job scheduling engine acquires database-level coordination locks while querying these indexes to determine which jobs to dispatch. As the index lookups grew slower, they began exceeding the database's configured statement timeout. When a lookup was canceled by the timeout, the scheduling process responsible for that work crashed and restarted. With no schedulers running, no workflow steps could be dispatched and all in-progress workflow executions became stuck. The database server itself remained healthy throughout the incident, with normal CPU and connection levels. The problem was exclusively lock and latency contention within the scheduling layer. A service restart cleared the crashed processes and temporarily restored execution. However, because the index bloat was still present, the same degradation recurred once query load resumed. A manual index rebuild fully restored performance and resolved the issue. ## Actions to Prevent Recurrence 1. **Automated index maintenance added** – Scheduled automatic index maintenance has been configured for the affected indexes. This ensures bloat cannot accumulate over time and eliminates the conditions that triggered this incident. 2. **Legacy indexes removed** – Unused legacy database indexes have been identified and removed, reducing the overall maintenance surface and simplifying future index hygiene. 3. **Monitoring coverage updated** – Our monitoring landscape is being reviewed and updated to reflect the current state of the workflow engine. This work will close gaps that allowed the degradation to go undetected before the first customer report.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.