Delay in Message Status Updates

What to do

• Check whether your product depends on Courier. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 0m it lasted.

  Anything that failed between 2024-05-28 16:34 UTC and 2024-05-20 19:30 UTC may need a replay.

• Subscribe to Courier's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=courier works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Courier 1

1. resolvedMay 20, 2024, 7:30 PM UTC

   On 2024-05-20 12:40 GMT-7, Courier experienced a sudden spike in outbound message volume. All messages were sent normally. However, the queue used to process message update events became overwhelmed and could not accept events at the rate they were produced. This caused a delay in message status updates as the queue backed up. Although the queue would have recovered eventually on its own, the engineering team chose to increase queue capacity to resolve the issue more quickly. This increase was implemented at 14:43, with full recovery of enqueued message update events by 14:50. Messages processed between 12:40 and 14:43 experienced a delay in status updates of up to 400 seconds, with a typical delay of about 100 seconds. There was no delay in message processing or delivery; all message update events were eventually processed. Outbound webhooks, which depend on the impacted queue, were similarly delayed, as were message statuses shown in the Logs UI and reported by the API.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.