- Level
- confirmed
- Worst impact
- Degraded
- Components
- Edge API
- Down Index
- 32of 100
What to do
Check whether you use Flagsmith Edge API. Check your stack
Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.Confirm your retries, queues and scheduled jobs caught up after the 1h 10m it lasted.
Anything that failed between 2024-01-18 14:40 UTC and 2024-01-18 15:50 UTC may need a replay.Subscribe to Flagsmith's feed to hear about the next one first. RSS feed
https://down.guide/feed?provider=flagsmith works in Slack (/feed subscribe), Discord feed bots and any RSS reader.
Updates from Flagsmith 11
- investigating
We are currently investigating this issue.
provider
- identified
The issue has been identified and a fix is being implemented.
provider
- monitoring
A fix has been implemented and we are monitoring the results.
provider
- investigating
Issues are still persisting for integrations using the Go client. We are investigating further.
provider
- investigating
We are continuing to investigate this issue.
provider
- identified
We have identified the remaining issue and are implementing a fix. ETA for full resolution: 15 minutes.
provider
- identified
We have rolled back in certain affected regions and have completed the work for the permanent fix. This is in the final stages of testing now and will be rolled out imminently.
provider
- identified
We are continuing to work on a fix for this issue.
provider
- identified
We are deploying the permanent fix now.
provider
- resolved
This incident has been resolved.
provider
- postmortem
### Timeline At around 13.45 today, we deployed a change to resolve a validation issue that had been introduced in a release earlier today. This validation issue affected only requests which provided a numeric value for the identity identifier. The new validation which was added, however, caused an issue for certain integrations since it also added a requirement for the traits key to be provided \(and not omitted\) which is not the case in some of our clients \(the Go client for example omits the traits key if the list is empty\). This meant that valid requests from these clients for identities with no traits were being incorrectly rejected as invalid. Once we received alerts for this from our monitoring and some of our affected customers we began investigating. At 14:54 we deployed a change which resolved the validation issue for certain cases, however not all. As such, at 15:06 we made the decision to roll back the affected regions, and at 15:48 we deployed a permanent fix for this including additional test cases to cover this behaviour. ### Impact Since the requests that were affected by this issue were those that had no traits, the impact was fairly limited and no trait data has been lost. Some identities will not have been created during this period, however, due to the nature of the Flagsmith integration, subsequent calls to identify those users will create them. ### Next Steps We have been working hard already on improving our release process for the Edge API. The first step of this, which is due to be released next week, is to improve our automated releases to rollback based on a number of additional alerting factors, including more granular looks at our error rates. This will ensure that, in future, a small subset of errors like this will trigger an immediate automated rollback. The next step after this is to create a more comprehensive end to end testing suite which exercises each of our SDKs to verify that the integrations are all compatible with any new changes.
provider
Reports count in aggregate; nothing about you is shown.
Corrections
None so far.
Something wrong on this page? Report an error or send us a note. Every fix is listed here.
Questions about DWN-725eef7859
Is Flagsmith still down?
No, this incident was resolved 2024-01-18 15:50 UTC. For Flagsmith's live state, see https://down.guide/flagsmith.
How long did it last?
1h 10m, from 2024-01-18 14:40 UTC to 2024-01-18 15:50 UTC, by the times Flagsmith posted.
What was affected?
Flagsmith named Edge API.
What caused it?
Flagsmith hasn't published a root cause. If it does, it shows here.
Where does this come from?
Flagsmith's official status page, https://status.flagsmith.com, quoted verbatim. down.guide's own probes, reports and social counts are labeled as ours.