Core API: Increased error rates

What to do

• Check whether you use Flagsmith Admin API. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 8m it lasted.

  Anything that failed between 2022-07-01 11:31 UTC and 2022-07-01 11:40 UTC may need a replay.

• Subscribe to Flagsmith's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=flagsmith works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Flagsmith 3

1. investigatingJul 1, 2022, 11:31 AM UTC

   We are seeing increased 502 responses to our Core API. We are aware of the cause and working on a fix. The Edge API is unaffected.

   provider

2. resolvedJul 1, 2022, 11:40 AM UTC

   This incident has been resolved.

   provider

3. postmortemJul 1, 2022, 11:40 AM UTC

   As part of a new feature rollout, there was a large database migration that needed to take place. We knew that the migration would take some time, however, it should not have affected production traffic. Unfortunately, despite our health check returning unhealthy until all migrations are complete, AWS ECS promoted the new version of the API application before the migrations were complete. This meant that the code that was running was expecting certain columns / data to be available in the database which weren’t there yet. We are still investigating what caused ECS to promote the new version before the migrations were complete.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.