- Level
- confirmed
- Worst impact
- Degraded
- Regions
- eu, us
What to do
Check whether you use Phrase in eu, us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.
Anything that failed between 2025-06-27 00:00 UTC and 2025-06-27 00:00 UTC may need a replay.Subscribe to Phrase's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=phrase works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Phrase 2
- resolved
The issue has been resolved.
provider
- postmortem
# **Root Cause Analysis** July 2, 2025 ### **Introduction** We would like to share more details about the events that occurred between 2:00 AM June 27 and 6:40 AM CET on June 28, 2025, which led to missed execution triggers and also a delay relative to typical workflow starting times. Missed execution triggers could be observed between: June 27, 03:37 PM CET to 03:55 CET and from June 27, 11:30 PM CET to June 28, 0:45 AM CET. Below is a timeline of events and the steps our engineering team is taking to prevent similar issues in the future. ### **Timeline** June 27, 2:00 AM CET: Monitoring showed high traffic spikes, which were still handled by the system gracefully. June 27, 03:37 PM CET: The traffic spikes increased and the engineering team received alerts that a system component had become unstable. Phrase engineers identified a large consumption in available memory, resulting in system instability. As a first mitigation the Orchestrator Workflow Engine component was restarted. June 27, 03:52 PM CET The component up again and continued to process scheduled workflows. June 27, 11:30 PM CET: The Orchestrator Workflow Engine component became overloaded again. New workflows were no longer being triggered. June 28, 12:04 AM CET: Phrase engineers took steps to halt a workload that was suspected as having consumed large amounts of memory. June 28, 12:45 AM CET: Phrase engineers applied further memory resources and the system continued again to trigger and process scheduled workflows. June 28, 6:40 AM CET: The backlog of workflows had been completely processed. **Root Cause** The incident was caused by sustained memory pressure on the Orchestrator Workflow Engine component, triggered by a spike in incoming events caused by a single workflow. This eventually prevented the system from processing new workflows. ### **Actions to Prevent Recurrence** A thorough review of the affected system’s resource configuration is being conducted. As part of this effort, the following improvements are implemented: * **Limit radius of impact of single workflows:** We will implement mechanisms to isolate resources to reduce the likelihood of cross workflow impact. * **Proactive detection:** We will introduce additional alerts to improve early detection, to allow us to take action before customers are impacted. * **Improved Scalability:** We are adding additional scaling mechanisms to key components of the Workflow Engine to handle sudden increases in workload. We are adjusting memory limits and requests for the impacted component to ensure greater stability under load.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-55e98ca733
Is Phrase still down?
No, this incident was resolved 2025-06-27 00:00 UTC. For Phrase's live state, see https://down.guide/phrase.
How long did it last?
0m, from 2025-06-27 00:00 UTC to 2025-06-27 00:00 UTC, by the times Phrase posted.
What was affected?
Phrase named in eu, us.
What caused it?
Phrase hasn't published a root cause. If it does, it shows here.
Where does this come from?
Phrase's official status page, https://status.phrase.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.