Incident Report

Background

Google’s back end SMTP Gmail delivery system allows customers accessing Gmail through third party email clients to send and receive email. To verify the identity of users, our SMTP delivery system uses an internal authentication framework to validate their account credentials before they are allowed to send email.

Summary

On Thursday, 26 January 2023 at 06:20 PT, some customers using third party email clients may have experienced errors and/or retry latency when attempting to send mail through our SMTP Gmail delivery system for a period of 1 hour and 48 minutes. We sincerely apologize to our Google Workspace customers who were impacted during this disruption. This is not the level of quality and reliability we strive to offer you. We have conducted an internal investigation and are taking immediate steps to improve the resiliency of our service.

Root Cause

After a detailed investigation, Engineers determined this issue was caused by memory limitations caused by unanticipated organic growth in one of the back end databases used by our internal authentication framework. This resulted in errors and/or latency experienced by customers using 3rd party email clients as they attempted to send emails to Gmail endpoint.

Remediation and Prevention

Google engineers were alerted to this issue through automated monitoring for errors and immediately began an investigation. At 06:45 PT, engineers determined that components of the authentication framework had run out of memory and started to scale up instances at 07:45 PT. The issue was completely resolved by 08:08 PT.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Deploying automatic scaling for unanticipated database growth.
• Deploying proactive detection and alerting capabilities for monitoring memory limits in our authentication framework.

We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On Thursday, 26 January 2023 between 06:20 and 08:08 PT, third party email clients connecting to Gmail’s SMTP services would have experienced errors or latency when attempting to send mail. Depending on the third party email client being used, this could have resulted in errors propagated to the end user or retries.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213 .

(All Times US/Pacific)

Incident Start:26 January 2023 06:20

Incident End:26 January 2023 08:08

Duration:1 hour, 23 minutes

Affected Services and Features:

Gmail

Regions/Zones:

Global

Description:

Gmail experienced mailing list errors and message delivery failures, due to the back end management system unavailability. Google will be completing a full Incident Report in the following days that will provide a detailed root cause.

Customer Impact:

• Some Gmail users observed their messages getting rejected.

The problem with Gmail has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Our team is continuing to investigate this issue. We will provide an update by Jan 26, 2023, 5:00 PM UTC with more information about this problem. Thank you for your patience. The affected users are able to access Gmail, but are seeing error messages, high latency, and/or other unexpected behavior.

We're investigating reports of an issue with Gmail. We will provide more information shortly.