# Api are having Performance degradation

- Provider: [Frontegg](https://down.guide/frontegg)
- Level: confirmed
- Impact: Degraded
- lasted 45m
- started 2022-08-14 23:54 UTC
- id: DWN-1138ea1ef1
- Official page: https://stspg.io/25nhvxr6qrmh

## Updates from Frontegg
### 2022-08-14 23:54 UTC: investigating
We are currently investigating this issue.

### 2022-08-15 00:00 UTC: investigating
We are continuing to investigate this issue.

### 2022-08-15 00:23 UTC: investigating
We are continuing to investigate this issue.

### 2022-08-15 00:24 UTC: monitoring
A fix has been implemented and we are monitoring the results.

### 2022-08-15 00:39 UTC: resolved
This incident has been resolved.

### 2022-08-16 12:05 UTC: postmortem
### \*\*Executive summary:\*\*
On August 15th, 2022 at 02:01 IST \\(UTC \\+2\\) Frontegg underwent a sophisticated DDOS subdomain organized attack. The attackers used multiple servers spread across a variety of Digital Ocean IPs. Each Server executed a low number of requests per second so our WAF did not trigger rate-limiting rules, yet it was recognized that many of the paths were related to WordPress engine's known weakness. 
By 03:21 the attack had been successfully mitigated. At 04:46 a second organized attack began. The restrictions put in place by the previous attack were helpful in mitigating the second attack. By 05:30 all traffic returned to normal
### \*\*Affect:\*\*
The incident caused a degraded performance to our API gateway. As a result, our API returned 504 and 524 errors to partial traffic over the course of the incident. The majority of these errors were returned between 02:01 IST and 02:30 IST, when our mitigation efforts began to have an effect. A majority of traffic was still able to go through without error during this time.
### \*\*Mitigation and resolution:\*\*
Our initial response to the attack was to increase our rate limiting and WAF constraints. This initial step was implemented at 02:30 IST. Once we understood the level of sophistication and distribution of the attack, we implemented changes on the application level, including a different routing mechanism and added more specific WAF constraints based on origins of the attacking traffic, which took effect by 03:21 IST.
### \*\*Preventive steps:\*\*
In order to prevent attacks like this in the future, we are implementing a more sophisticated route blocking mechanism to our API-gateway. Additionally we have reported the incident to the cloud provider which hosted a majority of the attacking traffic, and we are consulting with our WAF provider for further guidance on preventing such attacks.

---
down.guide reads providers' official status pages every minute and adds its own evidence, always labeled. Free, no account. Agents: MCP at https://down.guide/mcp, REST at https://down.guide/api/v1, feeds at https://down.guide/feed. Built by Sazabi (https://sazabi.com/?utm_source=down-guide&utm_medium=markdown).