Delayed Message Processing

What to do

• Check whether you use Courier API. Check your stack

  Drop a package.json, lockfile, Terraform or a list of domains at down.guide to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 4h 10m it lasted.

  Anything that failed between 2024-10-24 22:38 UTC and 2024-10-25 02:48 UTC may need a replay.

• Subscribe to Courier's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=courier works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Courier 8

1. investigatingOct 24, 2024, 10:38 PM UTC

   Courier has identified a delay in message processing for the messages API. We are currently investigating the root cause and will update you periodically.

   provider

2. investigatingOct 24, 2024, 11:01 PM UTC

   We are continuing to monitor the root cause of the delayed message processing.

   provider

3. investigatingOct 24, 2024, 11:29 PM UTC

   The team is still investigating and discussing the root cause internally. Messages are experiencing longer than normal processing times.

   provider

4. identifiedOct 25, 2024, 12:00 AM UTC

   The team has identified the issue and is rolling out a fix for the message delays.

   provider

5. identifiedOct 25, 2024, 12:40 AM UTC

   Fix has been deployed and will be live in around 1hr.

   provider

6. identifiedOct 25, 2024, 1:31 AM UTC

   The team identified an issue with the latest fix and reverted it. A new fix has been published to mitigate the message delay issue. ETA to land in production ~45 minutes.

   provider

7. monitoringOct 25, 2024, 2:43 AM UTC

   A fix has been implemented and delivery times are beginning to returning to expected levels. The team will continue to monitor.

   provider

8. resolvedOct 25, 2024, 2:48 AM UTC

   This incident has been resolved

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.