Message Logs Delays

What to do

• Check whether you use Courier Observability. Check your stack

  Drop a package.json, lockfile, Terraform or a list of domains at down.guide to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 55m it lasted.

  Anything that failed between 2024-10-21 15:56 UTC and 2024-10-21 18:51 UTC may need a replay.

• Subscribe to Courier's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=courier works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Courier 6

1. investigatingOct 21, 2024, 3:56 PM UTC

   The Courier team is investigating an issue with the event logger for message event logs hitting a bottleneck. The team is actively investigating. Messages are still sending.

   provider

2. investigatingOct 21, 2024, 3:56 PM UTC

   We are continuing to investigate this issue.

   provider

3. identifiedOct 21, 2024, 4:07 PM UTC

   The team is testing out a fix to reduce the bottlenecked log lines before releasing to production.

   provider

4. identifiedOct 21, 2024, 5:19 PM UTC

   The team encountered an issue with testing the fix and reverted the update. We are publishing a new update that should resolve the backlogged message logs.

   provider

5. monitoringOct 21, 2024, 6:30 PM UTC

   The fix has landed in production and the team is monitoring the message log queue. Message event logs should be flowing normally.

   provider

6. resolvedOct 21, 2024, 6:51 PM UTC

   The data stream is unblocked, and the message logs queue is resolved and flowing normally.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.