- Level
- confirmed
- Worst impact
- Major outage
- Components
- Translation center, Repo sync, OTA, Email delivery, Ordering, In-context editor, API
- Regions
- eu, us
What to do
Check whether you use Phrase Translation center, Repo sync, OTA, Email delivery and 3 more in eu, us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 45m it lasted.
Anything that failed between 2025-05-20 08:06 UTC and 2025-05-20 08:51 UTC may need a replay.Subscribe to Phrase's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=phrase works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Phrase 4
- investigating
Our engineers are currently investigating the root cause. We apologize for any inconvenience caused.
provider
- monitoring
The fix has been implemented and we are monitoring the results.
provider
- resolved
The issue has been resolved.
provider
- postmortem
### **Introduction** We would like to share more details about the events that occurred with Phrase between 09:52 AM CEST and 10:14 AM CEST on May 20, 2025, which led to a performance disruption of Strings EU and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 09:52 AM CEST: The Strings platform became unavailable due to a change in network configuration. Investigation began immediately. 10:14 AM CEST: A temporary fix was applied, restoring full platform availability. 03:43 PM CEST: A full fix addressing the underlying issue was deployed. **Root Cause** The issue was caused by a network configuration change introduced as part of a routine Kubernetes infrastructure upgrade. After the upgrade, some application components failed health checks, which led to automatic restarts and unavailability - even though the services themselves were actually running. This happened only in cases where enhanced network security rules were applied to individual parts of the system. The way the Kubernetes and underlying network handled traffic in this setup led to health checks not working as expected. The problem was resolved by adjusting the network settings on the underlying infrastructure network stack to support these health checks properly in this configuration. **Actions to Prevent Recurrence** We have updated our infrastructure configuration to always include this network setting going forward. This ensures that future upgrades will not trigger the same issue. We will also plan and roll out future Kubernetes upgrades in a more granular manner, targeting smaller segments of infrastructure first and performing changes outside of peak traffic hours. This will help us reduce the risk of platform-wide impact and detect potential issues earlier in the rollout process.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-0776560c07
Is Phrase still down?
No, this incident was resolved 2025-05-20 08:51 UTC. For Phrase's live state, see https://down.guide/phrase.
How long did it last?
45m, from 2025-05-20 08:06 UTC to 2025-05-20 08:51 UTC, by the times Phrase posted.
What was affected?
Phrase named Translation center, Repo sync, OTA, Email delivery and 3 more in eu, us.
What caused it?
Phrase hasn't published a root cause. If it does, it shows here.
Where does this come from?
Phrase's official status page, https://status.phrase.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.