Incident Report

Summary

On Tuesday, 17 January 2023 23:45 US/Pacific, Google Meet experienced an issue in which some users trying to watch livestreams globally received errors for 6 hours and 50 minutes. We have conducted an internal investigation and are taking steps to improve our service.

Root Cause

The investigation into this incident revealed that a recent rollout to the livestream player affected the playback mechanism, impacting Google Meet users on desktop web clients (except for Safari).

Google engineers investigated the code changes in the rollout submitted on 17 January 2023 23:45 US/Pacific and confirmed that it directly affected the Google Meet and YouTube integrations required for livestream functionality.

The issue caused Meet users to receive an error “This video format is not supported” when the Meet livestreams were accessed on desktop web clients. The issue did not affect Meet users who accessed the livestreams on iOS or Android device platforms and on Safari browsers.

Remediation and Prevention

Google engineers were alerted to the issue via an automated alert for viewer playtime on 18 January 2023 at 03:46 US/Pacific and immediately started an investigation. They identified the specific rollout affecting the livestream feature. The roll back of the identified code changes were completed on 18 January 2023 at 06:35 US/Pacific and successfully mitigated customer impact.

Google is committed preventing a future recurrence repeat of this issue by taking following actions:

• Launch system probers that regularly test the data pathway and collect configuration analysis.
• Create a centralized integration behavior model to help prevent Meet inconsistencies.
• Implement monitoring automation, including a metric dashboard to filter data for proactive alert triggering, identifying failures and discrepancies, ensuring alerting quality.

We would like to apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

Detailed Description of Impact

On 17 January 2023 from 23:45 US/Pacific to 18 January 2023 06:35 US/Pacific Google Meet users had the following impact:

• Users were unable to connect to livestreams from desktop web clients, except for Safari.
• Users trying to connect to livestreams from desktop web clients received the error message - "This video format is not supported."
• Users were able to connect to livestreams on iOS and Android devices.

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213 .

(All Times US/Pacific)

Incident Start:17 January 2023 23:45

Incident End:18 January 2023 06:35

Duration:6 hours, 50 minutes

Affected Services and Features:

Google Meet - Live stream

Regions/Zones:Global

Description:

Google Meet experienced an issue in which users trying to watch livestreams received errors for 6 hours and 50 minutes. From preliminary analysis, the root cause of the issue was identified as a recent rollout to the live stream player. Google will be completing a full IR in the following days that will provide a full root cause.

Customer Impact:

• Users who accessed Meet livestreams from clients other than iOS, Android, or Safari received an error “This video format is not supported” instead of a stream.

The problem with Google Meet has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Our team is continuing to investigate this issue. We will provide an update by Jan 18, 2023, 3:00 PM UTC with more information about this problem. Thank you for your patience. The affected users are able to access Google Meet, but are seeing error messages, high latency, and/or other unexpected behavior.

We are continuing to investigate this issue. We will provide an update by Jan 18, 2023, 2:50 PM UTC detailing when we expect to resolve the problem. The affected users are able to access Google Meet, but are seeing error messages, high latency, and/or other unexpected behavior.

Our team is continuing to investigate this issue. We will provide an update by Jan 18, 2023, 2:45 PM UTC with more information about this problem. Thank you for your patience. The affected users are able to access Google Meet, but are seeing error messages, high latency, and/or other unexpected behavior.

We're investigating reports of an issue with Google Meet. We will provide more information shortly. The affected users are able to access Google Meet, but are seeing error messages, high latency, and/or other unexpected behavior.