Incident Report

Summary

Incident Start:25 August 2025 09:25 US/Pacific

Incident End:25 August 2025 13:25 US/Pacific

Duration:4 hours

On Monday, 25 August 2025, Google Voice customers experienced intermittent dropped calls and connection issues globally for a duration of 4 hours.

To our affected customers whose business was impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

A software update inadvertently caused a massive number of devices to simultaneously connect to a critical component of our voice infrastructure. This surge of connection attempts overwhelmed the component, resulting in server tasks dropping traffic and repeatedly crashing for approximately 4 hours.

The issue was compounded because the devices were configured to immediately and repeatedly retry failed connections, creating a feedback loop that intensified the overload.

As a result, this core component was unable to process traffic from other clients, including Google Voice. This resulted in intermittent dropped calls and a significant percentage of call failures during the incident’s peak.

Remediation and Prevention

Google engineers were alerted to the issue via an internal monitoring at 07:20 US/Pacific on 25 August and immediately started an investigation. Once the nature and scope of the issue was identified, Google engineers rolled back the software update and increased capacity to the affected servers.These actions successfully mitigated the issue for all affected users by 13:25 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• To prevent service degradation, client applications implement a longer retry backoff upon receiving an HTTP 503 error for connection failures.
• Improve and update software rollout procedures and tooling to protect against unintentional changes and resulting traffic spikes.
• Improve capacity planning and server utilization monitoring for the affected servers.

We apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

Detailed Description of Impact

From 09:25 to 13:25 US/Pacific on 25 August 2025, Google Voice customers experienced intermittent dropped calls and connection issues globally for a duration of 4 hours.

Both enterprise and consumer users experienced intermittent dropped calls and connection issues across all devices such as mobile, web, and desk phones. At the peak of the incident, call failure rates reached up to 25%, with approximately one call dropped per second due to peer timeouts.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213 .

(All Times US/Pacific)

Incident Start:25 August 2025 09:25

Incident End:25 August 2025 13:25

Duration:4 hours

Affected Services and Features:

Google Voice

Regions/Zones:Global

Description:

Google Voice customers experienced intermittent dropped calls and connection issues globally for a duration of 4 hours. Based on preliminary analysis, the root cause is a software rollout which resulted in increased traffic and overloaded key infrastructure Google uses for voice calls.

Customer Impact:

Affected customers intermittently experienced dropped calls and connection issues. This issue may have affected our enterprise and consumer account customers across all devices (mobile, web, deskphones or dedicated hardware).

Summary:

Google Voice customers may have been unable to place or receive calls on their Google Voice clients and deskphones

Description:

The issue with Google Voice has been resolved for all affected users as of Monday, 2025-08-25 12:57 PDT

The issue was due to a hardware device firmware update causing higher than expected traffic to an overloaded server. Our engineers have mitigated the issue by disabling the software change and adding extra capacity to the affected servers.

We thank you for your patience while we worked on resolving the issue.

Diagnosis / Customer Symptoms:

Affected customers may have experienced dropped calls and connection issues intermittently.

Workaround:

None at this time.

Summary:

Google Voice customers may have been unable to place or receive calls on their Google Voice clients and desk phones

Description:

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2025-08-25 15:00 PDT.

Diagnosis / Customer Symptoms:

Affected customers may have experienced dropped calls and connection issues intermittently.

Workaround:

None at this time.