Increased error rate sending and receiving messages on email-based channels

What to do

• Check whether you use Front SMTP (non-Gmail / O365), Gmail, O365 in eu, eu-west-1, us, us-west-1. Check your stack

  Drop a package.json, lockfile, Terraform or a list of domains at down.guide to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 6h 26m it lasted.

  Anything that failed between 2024-09-27 20:25 UTC and 2024-09-28 02:51 UTC may need a replay.

• Subscribe to Front's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=front works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Front 8

1. investigatingSep 27, 2024, 8:25 PM UTC

   We are currently investigating this issue.

   provider

2. investigatingSep 27, 2024, 8:25 PM UTC

   We are continuing to investigate this issue.

   provider

3. identifiedSep 27, 2024, 8:38 PM UTC

   We have identified the cause of the issue and are working on a fix. The issue affects some but not all channels.

   provider

4. identifiedSep 27, 2024, 9:02 PM UTC

   We are validating the fix in our staging environment.

   provider

5. identifiedSep 27, 2024, 9:27 PM UTC

   We have validated the fix and are in the process of doing a manual deploy to circumvent the current Docker outage.

   provider

6. monitoringSep 27, 2024, 9:47 PM UTC

   The fix has been deployed and we are seeing email delivery services have recovered. We're monitoring and catching up on backlog so there might be residual delays.

   provider

7. monitoringSep 28, 2024, 2:21 AM UTC

   Customers in us-west-2 and eu-west-1 regions have fully recovered. We are still working through the incoming message backlog on us-west-1.

   provider

8. resolvedSep 28, 2024, 2:51 AM UTC

   This incident has been resolved.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.