Widespread Internet Issue

What to do

• Check whether you use Braze Dashboard, Dashboard, Dashboard, Dashboard and 47 more in eu, us. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 28m it lasted.

  Anything that failed between 2025-06-12 19:01 UTC and 2025-06-12 21:30 UTC may need a replay.

• Subscribe to Braze's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=braze works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Braze 10

1. investigatingJun 12, 2025, 7:01 PM UTC

   There appears to be a widespread internet issue affecting multiple services around the internet. Cloudflare, a Braze sub-processor, is having an incident (https://www.cloudflarestatus.com/incidents/25r9t0vz99rp) and Google Cloud is having a major outage (https://status.cloud.google.com/). This is affecting our ability to deliver push notifications via Google Firebase. AWS is not provisioning new servers, which is preventing our ability to scale up, and may be related to this overall internet issue. We will provide additional updates soon.

   provider

2. identifiedJun 12, 2025, 7:52 PM UTC

   The Google Cloud outage is preventing our ability to scale up our kubernetes clusters. As a result, customers will experience message sending and data processing latency until this recovers. We are also seeing SDK API errors across multiple clusters.

   provider

3. identifiedJun 12, 2025, 8:30 PM UTC

   Multiple Braze Services are experiencing degradation due to the aforementioned incident with Cloudflare and Google Cloud Platform. While all services are available and operating correctly, we are unable to scale up our infrastructure and have taken some precautions to prevent a cascading failure once the Google and Cloudflare incidents resolve and traffic starts flowing at higher rates again. Our SDK API is mostly returning error response codes to SDKs–this directs them to retry requests at a later time to throttle processing demand. Events and attribute updates will not be lost, and our SDKs will periodically retry requests automatically. Our REST APIs are operational; however, as inbound internet traffic is challenged, some requests may have timed out before they get to Braze. Per standard practice, any REST API calls that timeout or receive a 50X response code should be retried. Dashboard users may experience sporadic login timeouts when visiting https://dashboard.braze.com until upstream providers recover. Outbound messaging and data processing are continuing, but may experience latency until the incident is fully resolved upstream and Braze can re-scale our infrastructure. We are starting to see some signs of recovery and will be proactively ramping services back to normal behavior as we are able to. We will also continue to post regular updates.

   provider

4. identifiedJun 12, 2025, 8:38 PM UTC

   Our SDK API has resumed normal processing. Events and attribute updates will not be lost; we now are automatically retrying and processing those events. Dashboard login services have now recovered, and users can log in through https://dashboard.braze.com.

   provider

5. identifiedJun 12, 2025, 8:43 PM UTC

   We are now actively scaling up our infrastructure and expect to return to normal performance soon. As this happens, the Outbound Messaging and Data Processing backlog is being worked on. We expect latency to dissipate as we process through the queue.

   provider

6. identifiedJun 12, 2025, 8:46 PM UTC

   We continue to see signs of recovery and are ramping services back to normal as we are able to. Our REST APIs are operational; however, as some inbound internet traffic remains challenged, some requests may have timed out before they get to Braze. Per standard practice, any REST API calls that timeout or receive a 50X response code should be retried.

   provider

7. identifiedJun 12, 2025, 8:58 PM UTC

   Our infrastructure has fully scaled up, so we are processing well above our typical speeds to work through the backlogs. Several of our clusters are now fully operational, as the Data Processing and Outbound Messaging backlogs have been fully processed. US-01, US-02, US-03, US-04, US-05, US-07, and EU-02 are fully recovered.

   provider

8. identifiedJun 12, 2025, 9:07 PM UTC

   Google has confirmed that Android Push / Firebase messaging is fully operational, and backlogs have been processed.

   provider

9. identifiedJun 12, 2025, 9:13 PM UTC

   US-06 is now fully recovered. Data Processing and Outbound Messaging backlogs have been fully processed.

   provider

10. resolvedJun 12, 2025, 9:30 PM UTC

    EU-01 is now fully recovered. Data Processing and Outbound Messaging backlogs have been fully processed. With that, Braze is fully operational.

    provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.