Small number of intermittent requests to Edge API failing

What to do

• Check whether you use Flagsmith Edge API in us, us-east-2. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2d 18h it lasted.

  Anything that failed between 2024-12-16 19:07 UTC and 2024-12-19 14:04 UTC may need a replay.

• Subscribe to Flagsmith's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=flagsmith works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Flagsmith 9

1. investigatingDec 16, 2024, 7:07 PM UTC

   We are currently investigating the Edge API sporadically timing out some requests or refusing connections.

   provider

2. investigatingDec 16, 2024, 7:10 PM UTC

   The issue seems to only be affecting clients connecting to the us-east-2 region. We're currently redirecting traffic away from the region.

   provider

3. monitoringDec 16, 2024, 8:17 PM UTC

   We have migrated all traffic away from us-east-2 and are monitoring the impact.

   provider

4. investigatingDec 16, 2024, 9:59 PM UTC

   We have had further reports of similar issues following the migration of traffic away from us-east-2, we are continuing to investigate.

   provider

5. investigatingDec 16, 2024, 11:43 PM UTC

   We have passed on all of the troubleshooting information to our infrastructure provider and are awaiting further information from them on this issue. If you are affected by this issue, please get in touch with us at support@flagsmith.com with any information that you can share so we can identify this intermittent issue.

   provider

6. identifiedDec 17, 2024, 3:49 PM UTC

   The issue has been isolated to a specific component in our hosting provider's network infrastructure. We are working with them, and escalating to get the issue resolved.

   provider

7. monitoringDec 18, 2024, 6:56 PM UTC

   Our infrastructure provider has confirmed they were intermittently unavailable between December 14, 01:00 UTC and December 17, 22:50 UTC. We are working with affected customers to confirm that this issue is resolved.

   provider

8. resolvedDec 19, 2024, 2:04 PM UTC

   We have received confirmation from multiple customers in the US east region now that the issue has been resolved.

   provider

9. postmortemDec 20, 2024, 5:20 PM UTC

   ## What Happened On Monday 16th December at around 14:25 GMT, we started receiving reports from customers that they were seeing elevated errors in their monitoring regarding connections to the Flagsmith API. Our investigation suggested that there were no application level issues in the Flagsmith platform. Since all customer reports were coming from Eastern US regions, we moved traffic away from the region, redirecting to our US west region. By this time, we were also able to carry out our own testing using infrastructure set up in US east. Our testing, and feedback from our customers showed that moving the traffic away did not resolve the issue. At 19:15 GMT, we opened a ticket with our infrastructure partner. Their investigations also confirmed that there were no issues with the Flagsmith platform itself. At 21:53 GMT, we created a ticket directly with our infrastructure provider \(AWS\) for them to investigate. At around 23:30 GMT on 16th December, we implemented a work around by creating a new DNS record to point directly to our infrastructure in US east, bypassing the latency based routing \(as provided by AWS Global Accelerator\). We shared this with customers experiencing the issues, and they confirmed that it resolved the issue. From here, we continued to investigate the issue with AWS support, providing them with additional information based on our testing, and reports from our customers. On 18th December at 17:58, we received the following information from AWS confirming that there had been an issue with Global Accelerator in the US East region. > The team confirmed that between December 13 5:00 PM PST and December 17 2:50 PM PST, AWS Global Accelerator experienced intermittent connection failures for client traffic served by the Ashburn, Virginia edge location. The issue has been resolved and the service is operating normally. Following this, we were able to confirm that the issue was no longer reproducible for us, or for our customers. ## What’s Next? We have requested, and are currently waiting for, a full post-mortem from the AWS team which may affect our next steps. In the meantime, we have begun looking at alternative solutions for the Global Accelerator that we may be able to keep as a cold standby in case of similar issues in the future.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.