Incident Report

Summary

On Monday, 24 November 2025, Google Chat experienced an overload which led to service disruptions for users for a duration of 1 hour and 53 minutes. Customers encountered issues including the inability to send messages, load the chat application, and start new conversations. Some notifications were also incorrectly displayed with "Anonymous" as the sender.

We deeply regret the disruption our Google Workspace customers experienced. We are committed to improving our platform's reliability and performance, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The trigger of the outage was transient errors served from an internal microservice in Europe that caused Chat backend processes to execute a fallback path to retrieve this data. The root cause was that the data retrieved from the fallback path subsequently caused an update to Chat metadata which were sent to user devices, causing those devices to reload data from Chat backend servers, thereby causing more updates to be sent to user devices resulting in a cascading effect that overloaded Chat servers in Europe.

Google engineers have implemented a mitigation to limit high rates of requests to prevent this cascading loop from recurring.

Remediation and Prevention

Google engineers were alerted to the incident via alerts and immediately started an investigation.

To fully mitigate the impact, our engineering teams took the following steps:

1. Increased Server Capacity:To handle the unexpected surge in traffic, we significantly increased the capacity of our servers in the affected region to ensure the system could process all incoming requests smoothly.
2. Limited the Rate of Metadata Updates: To prevent the cascading loop of requests from overloading Chat servers again, Google engineers have limited the rate of metadata updates so that Chat servers will not be overloaded in this way.
3. Paused Non-Essential Background Tasks:To prioritize the restoration of core chat functions, we temporarily halted several internal background processes. This action helped to reduce the overall load on our systems, freeing up resources to serve user traffic more effectively.
4. Gradual Traffic Restoration:Our automated defenses had initially blocked some traffic to prevent a complete system overload. A crucial part of the recovery was to gradually and carefully allow this traffic back. This controlled approach ensured that as users reconnected, the system wouldn't be overwhelmed again, allowing everyone to successfully re-establish their connection to Google Chat without getting stuck in a retry loop.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Fix the immediate cause: Stop errors in calls to the microservices from triggering metadata updates to devices. The fix prevents a cascading loop of client devices making requests back to Chat servers resulting in overload.
• Prevent devices from sending excessive load: Implement changes in Chat web and mobile apps to prevent them from performing full reload of data from Chat servers when it is not needed. This reloading contributed greatly to the overload of Chat servers.
• Improve Monitoring: Enhance monitoring to identify which specific metadata updates to user devices cause reloads in order to more quickly mitigate similar issues in the future.

Detailed Description of Impact

On Monday, 24 November 2025 at 00:02 US/Pacific, Google Chat users in Europe experienced a service disruption affecting core features of the service for a duration of 1 hour and 53 minutes.

At its peak, the incident affected approximately 20.2% of users in Europe. The issue was resolved by 01:55 US/Pacific.

Users encountered problems with the following core functionality:

• Initial Application Load: Many users had trouble loading the Google Chat application.
• Opening Chat Rooms: Users experienced errors when trying to open existing chat rooms.
• Sending Messages: Users faced delays sending messages.
• I ncreased Latency: There was an increase in message delivery latency for affected users.
• Notification Rendering: Some notifications were also incorrectly displayed with "Anonymous" as the sender.

Preliminary Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213 .

Summary

On Monday, 24 November 2025, Google Chat experienced an overload which led to service disruptions for users for a duration of 1 hour and 53 minutes. Customers encountered issues including the inability to send messages, load the chat application, and start new conversations. Some notifications were also incorrectly displayed with "Anonymous" as the sender

Root Cause

The trigger of the outage was transient errors served from an internal microservice in Europe that caused Chat backend processes to execute a fallback path to retrieve this data. The root cause was that the data retrieved from the fallback path subsequently caused an update to Chat metadata which were sent to user devices, causing those devices to reload data from Chat backend servers, thereby causing more updates to be sent to user devices, resulting in a cascading effect that overloaded Chat servers in Europe.

Google engineers have implemented a mitigation to limit high rates of requests to prevent this cascading loop from recurring.

Remediation

Google engineers were alerted to the incident via alerts and immediately started an investigation.

To fully mitigate the impact, our engineering teams took the following steps:

1. Increased Server Capacity: To handle the unexpected surge in traffic, we significantly increased the capacity of our servers in the affected region to ensure the system could process all incoming requests smoothly.
2. Limited the Rate of Metadata Updates: To prevent the cascading loop of requests from overloading Chat servers again, Google engineers have limited the rate of metadata updates so that Chat servers will not be overloaded in this way.
3. Paused Non-Essential Background Tasks: To prioritize the restoration of core chat functions, we temporarily halted several internal background processes. This action helped to reduce the overall load on our systems, freeing up resources to serve user traffic more effectively.
4. Gradual Traffic Restoration: Our automated defenses had initially blocked some traffic to prevent a complete system overload. A crucial part of the recovery was to gradually and carefully allow this traffic back. This controlled approach ensured that as users reconnected, the system wouldn't be overwhelmed again, allowing everyone to successfully re-establish their connection to Google Chat without getting stuck in a retry loop.

Detailed Description of Impact

For a duration of 1 hour and 53 minutes, Google Chat users in Europe experienced a service disruption which manifested as an inability to use core features of the service, such as sending messages and starting conversations.

Summary

Google Chat users might not be able to open chat while connecting from Europe.

Description

The issue with Google Chat has been resolved for all affected users as of Monday, 2025-11-24 01:55 PST.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Symptoms

Google Chat users might not be able to open chat while connecting from Europe.

Workaround

None at this time.

Summary

Google Chat users might not be able to open chat while connecting from Europe.

Description

We are experiencing an issue with Google Chat beginning on Monday, 2025-11-24 00:02 PST.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2025-11-24 03:00 PST with current details.

We apologize to all who are affected by the disruption.

Symptoms

Google Chat users might not be able to open chat while connecting from Europe.

Workaround

None at this time.