Core API outage

What to do

• Check whether your product depends on Flagsmith. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2024-07-10 13:31 UTC and 2024-07-09 23:30 UTC may need a replay.

• Subscribe to Flagsmith's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=flagsmith works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Flagsmith 2

1. resolvedJul 9, 2024, 11:30 PM UTC

   Core API and admin dashboard outage on 10th July 2024.

   provider

2. postmortemJul 10, 2024, 1:31 PM UTC

   At around 23:35 UTC, July 9th we received an alert that our Core API was not responding. This resulted in our SaaS customers not being able to use the Flagsmith dashboard \([app.flagsmith.com](http://app.flagsmith.com)\). Customers SDK’s serving flags were not impacted for those using the Edge API. Please note, any customers still using our Core API to serve flags were also impacted. This number is limited as we have advised customers to migration to the Edge API starting in June 2022. Our team resolved the issue at 3:06 UTC, July 10th and the Core API was fully responsive. The root cause of the issue was a database running at maximum CPU caused by requests to an end point that triggered an inefficient query. We also had our load balancer consistently recycling unhealthy API tasks that also strained the system due to unnecessary database connections. These two items combined, resulted in the core API being unresponsive. We recovered the database by dropping all traffic and terminating all open connections. This allowed the database to be recovered and process traffic normally. We are mitigating future issues like this by doing the following: * Optimizing the query that was triggered that used too much CPU capacity \(note that this has been completed and deployed to our production SaaS environment\) * Add better alerting when inefficient queries are identified in the application * Improving our internal tools \(e.g. PagerDuty\) to improve response time of issue identification triggered by some team members being out of office

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.