# Frontegg Services are showing Degraded Performance in EU & US

- Provider: [Frontegg](https://down.guide/providers/frontegg)
- Confidence: Confirmed
- Impact: Partial outage
- lasted 3h 5m
- started 2023-05-31 13:08 UTC
- eu, us
- id: DWN-15ad142437
- Official page: https://stspg.io/n615f2rxr8yl

## Root cause
> However, the migration job didn't run due to an edge race condition in our CD infrastructure, causing the service to remain in a schema mismatch state.

Category: Deploy or config change. [Postmortem](https://stspg.io/n615f2rxr8yl)

## Updates from Frontegg
### 2023-05-31 13:08 UTC: investigating
We are currently investigating this issue.

### 2023-05-31 13:18 UTC: monitoring
A fix has been implemented and we are monitoring the results.

### 2023-05-31 16:13 UTC: resolved
This incident has been resolved.

### 2023-06-01 14:27 UTC: postmortem
Executive summary:
On Wednesday May 31st, 2023, at 12:55 GMT we deployed a minor version to one of our services. Shortly after at 12:56 GMT, Frontegg’s US monitoring system started sending alerts for an authentication service which was not performing as expected, and the team immediately began investigating the issue. 
At 13:01 GMT we started getting alerts from Frontegg’s EU monitoring as well regarding the same service, shortly after, we started to get complaints from customers.
At 13:04 GMT, 8 min after we started getting the alerts the team concluded that it was sourced by a recent change that was deployed. As part of the change, there was a database migration for one of our primary services. However, the migration job didn't run due to an edge race condition in our CD infrastructure, causing the service to remain in a schema mismatch state.
At this point we immediately started a rollback process for both EU & US regions that was completed by 13:16 GMT. Once the rollback completed, we noticed that our services are working as expected again and customers also reported that they were no longer experiencing issues.
‌
Affect:
Most requests to customers’ custom Frontegg domains resulted in 401/404 responses or inability to authenticate.
For the EU region - between 12:59 to 13:16 GMT time.For the US region - between 12:56 to 13:14 GMT time
Mitigation and resolution:
Following the monitoring alerts the incident response team immediately identified the potential corrupted service and started rollback procedure with the previous successful deployment.
Preventive steps:
\* We defined a gated process for deploying DB migration changes
\* A schema validation on service init to prevent schema mismatch cases was added
\* Will add deployment validation that will fail deployment if migration didn’t run
\* Will remove the high dependency in that specific service as a single-point-of-failure for the main system flows
\* Reduce service rollback time by running only relevant part of the CD pipeline

From the builder: does your app depend on Frontegg? [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=frontegg) detects problems in your own logs without hand-written monitors, and you can ask it in plain language which errors mention Frontegg.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by [Sazabi](https://www.sazabi.com/?utm_source=down-guide&utm_medium=markdown&utm_content=DWN-15ad142437), AI-native observability and security: automatic issue detection, plain-language log search and live dashboards.