Incident Report

Summary

On 10 October 2022 at 15:18 PT, Google was alerted to a subset of users experiencing issues with Duo phone number registration with our Google Meet iOS application. Once the root cause was identified, Google pushed out an update to the App Store on 4 November 2022. We believe most users (66%) updated their app by 7 November 2022. To our Google Workspace customers who were impacted during this outage, we sincerely apologize. We have conducted an internal investigation and are taking steps to improve our service.

Root Cause

The database version for our Google Meet backend infrastructure uses a specific library to complete the database operations to register or reregister a Duo calling account. An issue with the error-handling mechanism in the library resulted in the backend operations not functioning as intended. This caused affected users to be unable to register or reregister phone numbers, preventing the use of calling features, on the Meet iOS app.

Remediation and Prevention

Google engineers were alerted to the situation on 10 October 2022 at 15:18 and immediately started an investigation. Engineers reviewed the Google Meet for iOS app’s feedback logs and identified the root cause as related to registration workflow. Code improvements were identified to handle the app error seen when a user tries to register their account. The fix was deployed in a new version of the iOS app, which was rolled out to the App Store by 4 November 2022. At least 66% of users upgraded to the new version by 7 November 2022.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Deploy additional client logging to Meet to report registration-related issues, so we can respond in a timely manner.
• Deploy additional monitoring for registration errors reported by the client to more quickly alert us if these types of events happen in the future.

We apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

Detailed Description of Impact

From 10 October 2022 at 15:18 to 7 November 2022 at 08:20, Google Meet iOS app users may have encountered an “unable to add calling” error when trying to register or reregister their Duo calling account, which resulted in phone numbers not registering as intended.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213 .

(All Times US/Pacific)

Incident Start:28 October 2022 09:22

Incident End:04 November 2022 11:05

Duration:7 days, 1hour, 43 minutes

Affected Services and Features:

Google Meet - iOS App

Regions/Zones:Global

Description:

A subset of Google Meet iOS App users experienced issues with Phone Number registration. From preliminary analysis, the root cause is a latent issue at the time of registration. The latest version of Google Meet iOS App which contains the fix was released to the AppStore on 04 November 2022.

Customer Impact:

Google Meet iOS App users may have encountered an “unable to add calling” error which resulted in Phone number registration failures.

The problem with Google Meet has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

We expect to resolve the problem affecting a significant subset of users of Google Meet at Nov 4, 2022, 8:00 PM UTC . Please note that this time frame is an estimate and may change.

We expect to resolve the problem affecting a significant subset of users of Google Meet at Nov 4, 2022, 8:00 PM UTC . Please note that this time frame is an estimate and may change.

We're aware of a problem with Google Meet affecting a significant subset of users. We will provide an update by Oct 31, 2022, 2:00 AM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.