Incident Report

Summary

From 7 June 2023 at 13:00 to 13 June 2023 at 10:50 US/Pacific, Google Sheets users experienced issues fetching finance data using the built-in GOOGLEFINANCE() function. The total impact duration lasted 5 days, 21 hours, 50 minutes.

Root Cause

Google engineers identified a configuration change that unintentionally triggered this issue during a recent rollout. The change caused requests to return an “N/A” value, which ultimately impacted Sheets end users' ability to fetch data using the built-in function.

Remediation and Prevention

An internal user alerted Google engineers to the issue on 13 June 2023 at 03:35 US/Pacific. Google engineers quickly identified the change that was deployed to a backend component. To mitigate the issue, engineers rolled back the change, and services were fully restored 13 June 2023 at 10:50 US/Pacific.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Extend our monitoring capabilities to identify similar issues faster.
• Optimize our debugging tools to enable us to investigate similar issues faster.
• Validate our test suite to ensure similar issues are eliminated before being deployed to production systems.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213 .

(All Times US/Pacific)

Incident Start:08 June 2023 at 14:00

Incident End:13 June 2023 at 07:50

Duration:4 days, 17 hours, 50 minutes

Affected Services and Features:

Google Sheets - GOOGLEFINANCE function

Regions/Zones:Global

Description:

Google Sheets users experienced issues with fetching finance data using the built-in GOOGLEFINANCE function. From preliminary analysis, the root cause of the issue was a change to a backend component. The change caused the GOOGLEFINANCE requests to return N/A value. The issue was mitigated by rolling back the change.

Customer Impact:

Affected users would have experienced data returned by the GOOGLEFINANCE function alternating between correct value and #NA.

The problem with Google Sheets has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

We expect to resolve the problem affecting a majority of users of Google Sheets at Jun 13, 2023, 3:00 PM UTC . Please note that this time frame is an estimate and may change. The affected users are able to access Google Sheets, but are seeing error messages, high latency, and/or other unexpected behavior.

We expect to resolve the problem affecting a majority of users of Google Sheets at Jun 13, 2023, 2:00 PM UTC . Please note that this time frame is an estimate and may change. The affected users are able to access Google Sheets, but are seeing error messages, high latency, and/or other unexpected behavior.

We're aware of a problem with Google Sheets affecting a majority of users. We will provide an update by Jun 13, 2023, 2:00 PM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change. The affected users are able to access Google Sheets, but are seeing error messages, high latency, and/or other unexpected behavior.