- Level
- confirmed
- Worst impact
- Major outage
- Components
- Analytics, API, CAT web editor, Connectors, File processing, Machine translation, Project management, Term base, Translation memory
- Regions
- eu, us
What to do
Check whether you use Phrase Analytics, API, CAT web editor, Connectors and 5 more in eu, us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 23m it lasted.
Anything that failed between 2025-08-06 18:20 UTC and 2025-08-06 18:42 UTC may need a replay.Subscribe to Phrase's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=phrase works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Phrase 4
- investigating
We are investigating the issue.
provider
- monitoring
The incident has been resolved, we are monitoring the system.
provider
- resolved
The incident has been resolved.
provider
- postmortem
# **Root Cause Analysis** August 6, 2025 ### **Introduction** We would like to share more details about the events that occurred with Phrase between 19:45 PM CEST and 20:40 PM CEST on August 6, 2025 which led to a performance disruption of all our TMS services in the EU environment and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** Aug 6, 2025 @ 19:45 – The Platform and Application teams were alerted to database issues. Aug 6, 2025 @ 20:00 – Engineers identified that a majority of database replicas were unresponsive. TMS services performance disruption is confirmed. Aug 6, 2025 @ 20:00 – The first replica node was restarted. Aug 6, 2025 @ 20:05 – The restarted node appeared online, but a significant imbalance in resource usage was observed on the primary node as there were too few healthy nodes. Aug 6, 2025 @ 20:10 – The second replica node was restarted. Aug 6, 2025 @ 20:15 – Both replica nodes restarted but were unable to connect to the primary node. Aug 6, 2025 @ 20:15 – Customer complaints were reported, prompting escalation to the next level of engineering support. Aug 6, 2025 @ 20:25 – Investigation confirmed a higher than expected number of connections to the primary node. Aug 6, 2025 @ 20:35 – The maximum number of allowed connections on the primary node were increased, allowing both replicas to reconnect. Aug 6, 2025 @ 20:40 – Replica nodes completed resynchronization, restoring normal service. ### **Root Cause** A recent routine upgrade of the database replicas introduced an unexpected change in how system memory was allocated under high connection loads. While this behavior did not appear in test environments, in production it caused the replica nodes to run out of memory. As a result, they became unresponsive and failed to synchronize with the primary database which caused their removal from load-balancing while increasing the load on the primary node. **Actions to Prevent Recurrence** As an immediate action we upscaled our database cluster - We’ve significantly increased the capacity so the cluster can handle higher demand without running into memory limits. Optimized memory usage - We’ve reduced how much memory the database requires for normal operations, making it more efficient and less prone to memory starvation under high load. Reviewed how memory is allocated - We have fine-tuned memory allocation for our database cluster resulting in minimal fragmentation of the address space and improving memory allocation performance as well. Enhanced monitoring - We’ve added extra monitoring metrics that alert us earlier if any database memory issues start to appear. Improve resilience of the database layer - We’re working on updating configuration and limits of our database layer to be more resilient in similar situations to lower any possible impact. Improved testing processes - We’re working with our application teams to include the database in more realistic load testing in lower \(non-production\) environments. This will help us catch potential issues before they ever reach customers.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-b5a3738e1d
Is Phrase still down?
No, this incident was resolved 2025-08-06 18:42 UTC. For Phrase's live state, see https://down.guide/phrase.
How long did it last?
23m, from 2025-08-06 18:20 UTC to 2025-08-06 18:42 UTC, by the times Phrase posted.
What was affected?
Phrase named Analytics, API, CAT web editor, Connectors and 5 more in eu, us.
What caused it?
Phrase hasn't published a root cause. If it does, it shows here.
Where does this come from?
Phrase's official status page, https://status.phrase.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.