Rotate etcd and control plane boot disk encryption keys

Standard

This page shows you how to rotate the etcd and control plane boot disk encryption keys that you configured for GKE control plane authority. This page is intended for cluster administrators and Security engineers.

Before you read this page, familiarize yourself with the following GKE control plane authority concepts:

Plan key rotations

This page shows you how to rotate the following credential components in your control plane:

The encryption key for etcd and the control plane boot disks.
The encryption key for the etcd internal backup that Google Cloud uses for disaster recovery.

You can also rotate the customer-managed certificate authorities (CAs) and service account signing keys that you configure with GKE control plane authority. For more information, see Rotate customer-managed control plane CAs and signing keys .

Keys in Cloud Key Management Service (Cloud KMS) don't expire. If your organization has a security requirement for key rotation, manually rotate the keys. To minimize disruptions to running workloads, don't configure automatic key rotation for these keys.

When you rotate the encryption keys for your control plane boot disks and etcd, GKE creates a new disk that's encrypted using the new key. For regional clusters, this operation has no downtime because of the replicated control plane. For zonal clusters, the control plane becomes unavailable until the new disk is available.

Before you begin

Before you start, make sure that you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update .
Note: For existing gcloud CLI installations, make sure to set the compute/region property . If you use primarily zonal clusters, set the compute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location . You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Have an existing cluster that uses control plane boot disk and etcd encryption
Identify the project IDs of the following Google Cloud projects:
- Key project: the project that contains your Cloud KMS resources.
- Cluster project: the project that contains your GKE cluster.
To do the validation tasks on this page, verify that the DATA_READ Data Access audit log is enabled for the Cloud Key Management Service (KMS) API. For more information, see Enable Data Access audit logs .

Required roles and permissions

To get the permissions that you need to rotate your customer-managed CAs and keys, ask your administrator to grant you the following IAM roles:

Manage keys or key versions: Cloud KMS Admin ( roles/cloudkms.admin ) on your key project
Configure clusters to use new keys: Kubernetes Engine Cluster Admin ( roles/container.clusterAdmin ) on your cluster project

For more information about granting roles, see Manage access to projects, folders, and organizations .

You might also be able to get the required permissions through custom roles or other predefined roles .

Limitations

You can update your clusters to new versions of existing keys, but you can't update your clusters to use entirely new keys for these encryption operations.

Grant IAM roles to the GKE service agent

To rotate your encryption keys for boot disks and etcd, grant the Kubernetes Engine KMS Crypto Key User ( roles/container.cloudKmsKeyUser ) role on the encryption keys in your key project to the GKE service agent in your cluster project:

 gcloud  
kms  
keys  
add-iam-policy-binding  
 KCP_DISK_KEY_NAME 
  
 \ 
  
--keyring = 
 KEYRING_NAME 
  
 \ 
  
--location = 
 LOCATION 
  
 \ 
  
--member = 
 "serviceAccount:service- CLUSTER_PROJECT_NUMBER 
@container-engine-robot.iam.gserviceaccount.com" 
  
 \ 
  
--role = 
roles/container.cloudKmsKeyUser  
 \ 
  
--project = 
 KEY_PROJECT_ID

Replace the following:

KCP_DISK_KEY_NAME : the name of the encryption key for your control plane boot disks and etcd disks.
KEYRING_NAME : the name of the key ring that holds the key.
LOCATION : the Google Cloud location of the key ring. This must be the same as your cluster location. For a list of regions, filter for "Region" in the Cloud KMS locations table .
CLUSTER_PROJECT_NUMBER : the numerical project number of the cluster project.
KEY_PROJECT_ID : the project ID of your key project .

Rotate the key for etcd and control plane boot disks

Create a new primary key version for the boot disk and etcd encryption key:

 gcloud  
kms  
keys  
versions  
create  
 \ 
  
--key = 
 KCP_DISK_KEY_NAME 
  
 \ 
  
--keyring = 
 KEYRING_NAME 
  
 \ 
  
--location = 
 LOCATION 
  
 \ 
  
--project = 
 KEY_PROJECT_ID 
  
 \ 
  
--primary

It might take up to 10 minutes for the changes to propagate in the project.

Find the current patch version of the cluster control plane:
```
 gcloud  
container  
clusters  
describe  
 CLUSTER_NAME 
  
 \ 
  
--location = 
 LOCATION 
  
 \ 
  
--format = 
 'value(currentMasterVersion)' 
 
```
Replace the following:
- CLUSTER_NAME : the name of your GKE cluster.
- LOCATION : the location of the GKE cluster control plane, such as us-central1 .
Upgrade the cluster control plane to the same version that it already uses.

Caution: In zonal clusters, the control plane becomes unavailable until the upgrade operation completes. In Autopilot clusters and regional Standard clusters, the control plane remains available.
```
 gcloud  
container  
clusters  
upgrade  
 CLUSTER_NAME 
  
 \ 
  
--master  
 \ 
  
--location = 
 LOCATION 
  
 \ 
  
--cluster-version = 
 CONTROL_PLANE_VERSION 
  
 \ 
  
--project = 
 CLUSTER_PROJECT_ID 
 
```
Replace the following:
- CONTROL_PLANE_VERSION : the version of the cluster control plane from the output of the previous step.
- CLUSTER_PROJECT_ID : the project ID of your cluster project .

After the control plane upgrade operation completes, the cluster uses the new key version to encrypt your control plane boot disks and any etcd disks.

Rotate the etcd internal backup encryption key

This section shows you how to rotate the encryption key for the etcd internal backup, which Google Cloud uses for disaster recovery purposes. You can configure automatic rotation for this key, but we recommend that you perform manual rotation instead. After you rotate the key, the cluster uses the new key version to encrypt new etcd internal backups.

etcd internal backups use the primary key version and occur at the following intervals:

Every 30 minutes for zonal clusters.
Every 10 minutes for Autopilot clusters and regional Standard clusters.

After you rotate the key, don't immediately disable or destroy the previous key version. Wait until enough time has passed for GKE to create new etcd internal backups that use the new key version.

Create a new primary key version for the etcd internal backup encryption key:
```
 gcloud  
kms  
keys  
versions  
create  
 \ 
  
--key = 
 ETCD_BACKUP_KEY_NAME 
  
 \ 
  
--keyring = 
 KEYRING_NAME 
  
 \ 
  
--location = 
 LOCATION 
  
 \ 
  
--project = 
 KEY_PROJECT_ID 
  
 \ 
  
--primary 
```
Replace the following:
- ETCD_BACKUP_KEY_NAME : the name of the etcd internal backup encryption key.
- KEYRING_NAME : the name of the key ring that holds the key.
- LOCATION : the Google Cloud location of the key ring. This must be the same as your cluster location. For a list of regions, filter for "Region" in the Cloud KMS locations table .
- KEY_PROJECT_ID : the project ID of your key project .