- Level
- confirmed
- Worst impact
- Partial outage
- Regions
- eu, us
What to do
Check whether you use Frontegg in eu, us. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 3h 5m it lasted.
Anything that failed between 2023-05-31 13:08 UTC and 2023-05-31 16:13 UTC may need a replay.Subscribe to Frontegg's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=frontegg works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Frontegg 4
- investigating
We are currently investigating this issue.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- resolved
This incident has been resolved.
provider
- postmortem
### **Executive summary:** On Wednesday May 31st, 2023, at 12:55 GMT we deployed a minor version to one of our services. Shortly after at 12:56 GMT, Frontegg’s US monitoring system started sending alerts for an authentication service which was not performing as expected, and the team immediately began investigating the issue. At 13:01 GMT we started getting alerts from Frontegg’s EU monitoring as well regarding the same service, shortly after, we started to get complaints from customers. At 13:04 GMT, 8 min after we started getting the alerts the team concluded that it was sourced by a recent change that was deployed. As part of the change, there was a database migration for one of our primary services. However, the migration job didn't run due to an edge race condition in our CD infrastructure, causing the service to remain in a schema mismatch state. At this point we immediately started a rollback process for both EU & US regions that was completed by 13:16 GMT. Once the rollback completed, we noticed that our services are working as expected again and customers also reported that they were no longer experiencing issues. **Affect:** Most requests to customers’ custom Frontegg domains resulted in 401/404 responses or inability to authenticate. For the EU region - between 12:59 to 13:16 GMT time.For the US region - between 12:56 to 13:14 GMT time ### **Mitigation and resolution:** Following the monitoring alerts the incident response team immediately identified the potential corrupted service and started rollback procedure with the previous successful deployment. ### **Preventive steps:** * We defined a gated process for deploying DB migration changes * A schema validation on service init to prevent schema mismatch cases was added * Will add deployment validation that will fail deployment if migration didn’t run * Will remove the high dependency in that specific service as a single-point-of-failure for the main system flows * Reduce service rollback time by running only relevant part of the CD pipeline
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-15ad142437
Is Frontegg still down?
No, this incident was resolved 2023-05-31 16:13 UTC. For Frontegg's live state, see https://down.guide/frontegg.
How long did it last?
3h 5m, from 2023-05-31 13:08 UTC to 2023-05-31 16:13 UTC, by the times Frontegg posted.
What was affected?
Frontegg named in eu, us.
What caused it?
Frontegg hasn't published a root cause. If it does, it shows here.
Where does this come from?
Frontegg's official status page, https://status.frontegg.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.