We are experiencing an issue with Cloud SQL API instance mutations in us-central1
40 views
Skip to first unread message
Google Cloud Platform Status
Jan 29, 2021, 5:00:00 PM
1/29/21
to google-cloud...@googlegroups.com
Description: We are experiencing an issue with Cloud SQL API instance mutations in us-central1. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific.
Our engineering team continues to investigate the issue. Instance mutation operations (such as instances.create or instances.delete) as well as requests for certificates (sslCerts.createEphemeral) when using Cloud SQL Proxy are impacted. Existing connections to Cloud SQL instances should be unaffected.
The product team has identified a potential mitigation and is working to implement it. As part of the mitigation, all requests to instances delete and instances restart will receive an error.
We will provide an update by Friday, 2021-01-29 15:00 US/Pacific with current details.
Diagnosis: Customers impacted by this issue may see increased latency or errors when performing Instance mutation requests, such as create, delete and update. Connection to Cloud SQL instances may be disrupted if a new ephemeral cert is required by the Cloud SQL Proxy. Errors returned may be 502 or 503.
Workaround: None at this time. We believe other regions are not impacted at this time.
Our engineering team continues to investigate the issue. Instance mutation operations (such as instances.create or instances.delete) as well as requests for certificates (sslCerts.createEphemeral) when using Cloud SQL Proxy are impacted. Existing connections to Cloud SQL instances should be unaffected.
The product team has identified a potential mitigation and is working to implement it. As part of the mitigation, all requests to instances delete and instances restart will receive an error.
We will provide an update by Friday, 2021-01-29 15:00 US/Pacific with current details.
Diagnosis: Customers impacted by this issue may see increased latency or errors when performing Instance mutation requests, such as create, delete and update. Connection to Cloud SQL instances may be disrupted if a new ephemeral cert is required by the Cloud SQL Proxy. Errors returned may be 502 or 503.
Workaround: None at this time. We believe other regions are not impacted at this time.
Google Cloud Platform Status
Jan 29, 2021, 5:38:11 PM
1/29/21
to google-cloud...@googlegroups.com
Description: We are experiencing an issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific.
Our engineering team continues to investigate the issue. Instance mutation operations (such as instances.create or instances.delete) as well as requests for certificates (sslCerts.createEphemeral) when using Cloud SQL Proxy are impacted. Existing connections to Cloud SQL instances should be unaffected. New connections without Cloud SQL Proxy are also unaffected.
The product team has identified a potential mitigation and is working to implement it. We believe the change will be finished rolling out by 15:00 US/Pacific, we will be continuously monitoring the service to evaluate the effectiveness of the mitigation.
We will provide an update by Friday, 2021-01-29 15:30 US/Pacific with current details.
Our engineering team continues to investigate the issue. Instance mutation operations (such as instances.create or instances.delete) as well as requests for certificates (sslCerts.createEphemeral) when using Cloud SQL Proxy are impacted. Existing connections to Cloud SQL instances should be unaffected. New connections without Cloud SQL Proxy are also unaffected.
The product team has identified a potential mitigation and is working to implement it. We believe the change will be finished rolling out by 15:00 US/Pacific, we will be continuously monitoring the service to evaluate the effectiveness of the mitigation.
We will provide an update by Friday, 2021-01-29 15:30 US/Pacific with current details.
Google Cloud Platform Status
Jan 29, 2021, 6:31:13 PM
1/29/21
to google-cloud...@googlegroups.com
Description: The issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1 has partially recovered. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific. Partial recovery occurred at 14:54 US/Pacific
Our engineering team has rolled out a mitigation and partially resolved the issue. We believe most of the service has recovered. At this time we believe only instances.delete and instances.restart continue to be impacted. Some operations such as instances.patch which require an instance.restart may also be impacted. We are working on further mitigation to bring the service back to a nominal operating state.
We will provide an update by Friday, 2021-01-29 16:30 US/Pacific with current details.
Our engineering team has rolled out a mitigation and partially resolved the issue. We believe most of the service has recovered. At this time we believe only instances.delete and instances.restart continue to be impacted. Some operations such as instances.patch which require an instance.restart may also be impacted. We are working on further mitigation to bring the service back to a nominal operating state.
We will provide an update by Friday, 2021-01-29 16:30 US/Pacific with current details.
Google Cloud Platform Status
Jan 29, 2021, 7:28:57 PM
1/29/21
to google-cloud...@googlegroups.com
Description: The issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1 has partially recovered. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific. Partial recovery occurred at 14:54 US/Pacific
Our engineering team has rolled out a mitigation and partially resolved the issue. We believe most of the service has recovered. At this time we believe only instances.delete and instances.restart continue to be impacted. Some operations such as instances.patch which require an instance.restart may also be impacted. The product team has identified the root cause of the issue and has begun implementing a fix. We expect that the Cloud SQL API will remain in the current partially resolved state for several hours until the fix is fully verified and rolled out.We will provide an update by Friday, 2021-01-29 19:00 US/Pacific with current details.
Google Cloud Platform Status
Jan 29, 2021, 9:59:48 PM
1/29/21
to google-cloud...@googlegroups.com
Description: The issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1 has partially recovered. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific. Partial recovery occurred at 14:54 US/Pacific
Our engineering team has rolled out a mitigation and partially resolved the issue. We believe most of the service has recovered. At this time we believe only instances.delete and instances.restart continue to be impacted. Some operations such as instances.patch which require an instance.restart may also be impacted. The product team has identified the root cause of the issue and has begun implementing a fix.
The product team has started rolling out the potential fix. We are monitoring the effects of this rollout to determine if the fix has solved the issue.Our engineering team has rolled out a mitigation and partially resolved the issue. We believe most of the service has recovered. At this time we believe only instances.delete and instances.restart continue to be impacted. Some operations such as instances.patch which require an instance.restart may also be impacted. The product team has identified the root cause of the issue and has begun implementing a fix.
We will provide an update by Friday, 2021-01-29 20:00 US/Pacific with current details.
Google Cloud Platform Status
Jan 29, 2021, 10:45:49 PM
1/29/21
to google-cloud...@googlegroups.com
The issue with Cloud SQL API instance mutations and SQL Proxy operations has been resolved for all affected users as of Friday, 2021-01-29 18:45 US/Pacific.
All operations including instance restart and delete are healthy.
We thank you for your patience while we worked on resolving the issue.
All operations including instance restart and delete are healthy.
We thank you for your patience while we worked on resolving the issue.
Reply all
Reply to author
Forward
Search
Clear search
Close search
Google apps
Main menu
