Delays in Android push notifications

What to do

• Check whether you use Opsgenie Mobile Notification Delivery, Mobile Notification Delivery. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 3h 46m it lasted.

  Anything that failed between 2023-04-04 14:30 UTC and 2023-04-04 18:16 UTC may need a replay.

• Subscribe to Opsgenie's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=opsgenie works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Opsgenie 6

1. investigatingApr 4, 2023, 2:30 PM UTC

   We are investigating an issue with our Android push notifications that is impacting some of our notifications for Android. We will provide more details within the next hour.

   provider

2. monitoringApr 4, 2023, 2:34 PM UTC

   The issue has been identified as caused by an error on Firebase (https://status.firebase.google.com/incidents/9ZPv9faHLen8bzLVSaft). We continue to monitor the situation and send update within the next hour.

   provider

3. monitoringApr 4, 2023, 3:23 PM UTC

   Android push notification delivery is fully operational for now, but we are still monitoring the Firebase outage (https://status.firebase.google.com/incidents/9ZPv9faHLen8bzLVSaft).

   provider

4. monitoringApr 4, 2023, 6:13 PM UTC

   We are continuing to monitor for any further issues.

   provider

5. resolvedApr 4, 2023, 6:16 PM UTC

   This incident on Firebase has been resolved. Android push notifications are operational.

   provider

6. postmortemApr 26, 2023, 6:41 AM UTC

   ### **SUMMARY** On April 4, 2023, between 13:32 and 14:50 UTC, Atlassian customers using Opsgenie faced significant delays while receiving Android push notifications. This was caused by an incident in a third party messaging service, which is responsible for Android push notification delivery. This in turn affected our systems. The incident was immediately detected by our monitoring tools, our on-call engineers were paged, and at 14:50 UTC our systems recovered successfully. The total time to resolution was about 80 minutes. ### **IMPACT** The overall impact was between April 4, 2023, 13:32 - 14:50 UTC in Opsgenie_._ The incident only resulted in delays in Android push notifications only, and these notifications were delivered successfully after FCM service was restored and no data loss occurred. ### **ROOT CAUSE** The issue was caused by an incident in a third party messaging service, which is responsible for delivering push notifications to Android devices. ‌ **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. The impact was immediately caught by our monitoring tools, and the responsible team immediately started analysis of incident. We value transparency with our customers and will continue to notify you and take any necessary actions promptly during an incident. In order to handle degradation or outage of messaging channels, Opsgenie recommends that users configure multiple channels of message delivery - including push notifications, mobile SMS, phone calls, and email. In order to improve our response for the future, we will also be analyzing whether we can employ autoscaling solutions for our systems in case of an outage/high load related to one notification channel. We apologize to customers whose services were impacted during this incident. Thanks, Atlassian Customer Support

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.