Summary:

What to do

• Check whether you use Google Workspace Google Chat. Check your stack

  Paste a package.json, lockfile, Terraform or a list of domains at down.guide/stack to see every provider you depend on.

• Confirm your retries, queues and scheduled jobs caught up after the 2h 44m it lasted.

  Anything that failed between 2026-08-28 17:50 UTC and 2026-08-28 20:34 UTC may need a replay.

• Subscribe to Google Workspace's feed to hear about the next one first. RSS feed

  https://down.guide/feed?provider=google-workspace works in Slack (/feed subscribe), Discord feed bots and any RSS reader.

Updates from Google Workspace 3

1. service_informationAug 28, 2026, 7:25 PM UTC

   **Summary:** Google Chat users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view. **Description:** We are experiencing an intermittent issue with Google Chat Friday, 2026-08-28 10:25 PDT. Our engineering team has investigated the issue and are currently working on mitigating the issue. We will provide more information by Friday, 2026-08-28 14:15 PDT with current details. We apologize to all who are affected by the disruption. **Customer Symptoms:** Users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view **Workaround:** Users can refresh to see new messages

   provider

2. resolvedAug 28, 2026, 8:55 PM UTC

   **Summary** Google Chat users experienced an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view. **Description** The issue with Google Chat has been resolved for all affected users as of Friday, 2026-08-28 13:45 PDT. From preliminary analysis the issue was triggered due to a recent change which was rolled back to mitigate the impact. We thank you for your patience while we worked on resolving the issue. **Customer Symptoms** Users were experiencing an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view. **Workaround** None as the issue is resolved.

   provider

3. resolvedSep 4, 2026, 3:43 AM UTC

   ## Incident Report ## Summary On Friday, August 28, 2026, between 10:50 US/Pacific and 13:34 US/Pacific for a duration of 2 hours and 44 minutes, Google Chat users experienced delays in receiving real-time incoming messages within active conversation streams across web, desktop, and mobile applications. While users continued to receive notifications for new messages, the message contents did not automatically render in active conversation streams until the user manually refreshed the application or sent a reply message. To our valued Google Workspace customers, we are deeply sorry for the service interruption you experienced. We are taking immediate and decisive action to address the issue and strengthen our systems against future incidents. ## Root Cause The incident was triggered during a global database schema rollout. An incompatible database schema change caused database writes required for message delivery to start failing. Specifically, a mechanism for schema synchronization—designed to ensure high consistency between primary storage and live message delivery—encountered a rare race condition bug. This caused backend message dispatchers to attempt writes against an unready schema state, leading to write failures. Because live message delivery had an active dependency on these writes completing successfully before pushing events to user screens, the failure caused incoming messages to experience synchronization delays. Separate push notification services and core message storage operated independently and were unaffected, which is why users still received alerts for new messages while active conversation streams experienced delays. ## Remediation and Prevention The issue was detected through automated backend alerting on dispatch error rates, alongside corresponding customer reports. Google Engineering initiated parallel workstreams to restore service. * At 12:44 US/Pacific, Google Engineering directly mitigated the source of the issue by removing the dependency that live message delivery had on the failing database write. This allowed new incoming messages to bypass the failed write and immediately resume normal real-time delivery to user screens. * At 13:00 US/Pacific, Engineering halted and reverted the database schema change to its previous verified stable version, restoring full database compatibility. * By 13:34 US/Pacific, as queued messages were processed, real-time message synchronization delays returned to 0% across all customer domains. Engineering verified that full service health was restored with zero message loss. To prevent this from happening again, we are executing the following preventative actions: * **Hardened internal detection:** Strengthen automated validation and verification to cover this specific failure scenario across pre-production and internal rollout stages. This aims to catch and mitigate schema incompatibilities and synchronization edge cases internally before any customer traffic can be impacted in the future. * **Resilience Against Race Conditions:** Implement Architectural improvements to make the schema synchronization and message delivery architecture resilient to race conditions. This aims to ensure that background schema and config synchronization inconsistencies cannot disrupt real-time message delivery. ## Detailed Description of Impact The incident impact window occurred between 10:50 US/Pacific and 13:34 US/Pacific on Friday, 28 August 2026\. This issue applied globally to all Google Chat environments. **Google Chat:** * **Delayed message rendering:** New messages did not appear in active conversation streams automatically. Users were required to manually refresh their client or send a new message/react to force the stream to synchronize. * **Stalled read states and inbox sorting:** Background actions, such as updating unread badges and moving active spaces to the top of the inbox, experienced heavy internal error rates and were significantly delayed for users. * **Search inde

   provider

Corrections

None so far.

Something wrong on this page? Report an error or send us a note. Every fix is listed here.