MURAL Notifications Delay

What to do

• Check whether you use Mural Notifications. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 5h 20m it lasted.

  Anything that failed between 2022-06-17 22:37 UTC and 2022-06-18 03:57 UTC may need a replay.

• Subscribe to Mural's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=mural works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Mural 9

1. investigatingJun 17, 2022, 10:37 PM UTC

   We're experiencing a service disruption that is causing notifications, including password reset emails, to not be delivered. We're investigating the issue and will restore regular service ASAP. Please check our status page for the most up-to-date info 👉 status.mural.co/

   provider

2. identifiedJun 17, 2022, 10:59 PM UTC

   The issue has been identified and our team is working on fixing it. We'll update this message as soon as we have more details.

   provider

3. identifiedJun 17, 2022, 11:48 PM UTC

   We have identified the path to return to full service and are working towards this as quickly as possible.

   provider

4. identifiedJun 18, 2022, 1:04 AM UTC

   We're continuing to work on resolving this issue.

   provider

5. identifiedJun 18, 2022, 2:01 AM UTC

   Our team is still working on resolving this issue. We'll update this message as soon as we have more details.

   provider

6. monitoringJun 18, 2022, 2:51 AM UTC

   We've implemented a fix that should resolve the issue. We're monitoring the results to make sure this has the intended effect and will update this message as soon as we've been able to confirm this.

   provider

7. monitoringJun 18, 2022, 3:53 AM UTC

   We are continuing to monitor the results to make sure the fix has the intended effects.

   provider

8. resolvedJun 18, 2022, 3:57 AM UTC

   The fix we implemented earlier has been successful in resolving the issue. We apologize for the inconvenience this interruption caused and will be conducting a full review to avoid a repeat in the future.

   provider

9. postmortemJun 24, 2022, 2:30 PM UTC

   **What happened?** On Friday, June 17 2022 at 22:20 UTC, our email service provider incorrectly identified a security risk with MURAL’s account. As a precaution, our email service provider suspended MURAL’s account. This resulted in MURAL product emails \(for example: invitations to join a mural\) not sending. MURAL worked with our email service provider to restore service as quickly as possible. Together with our service provider we implemented a fix at 02:50 UTC on June 18th, which restored full service and allowed the pending emails to be sent. We continued to monitor the results of the fix until 04:00 UTC, at which time the incident was closed. ‌ **Summary** MURAL’s platform remained available throughout this incident. There was a delay of 2 hours and 50 minutes sending product emails, but this did not impact service availability.

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.