Customizing node system configuration

This document shows you how to customize your Google Kubernetes Engine (GKE) node configuration by using a configuration file called a node system configuration .

A node system configuration is a configuration file that provides a way to adjust a limited set of system settings. In your node pool, you can use a node system configuration to specify custom settings for the kubelet Kubernetes node agent and for sysctl low-level Linux kernel configurations.

This document details the available configurations for a node system configuration and how to apply them to your GKE Standard node pools. Note that because GKE Autopilot clusters have a more managed node environment, their direct node system configuration options are limited compared to GKE Standard node pools.

Why use node system configurations

Node system configurations offer the following benefits:

• Performance tuning:optimize network stack performance, memory management, CPU scheduling, or I/O behavior for demanding applications like AI training or serving, databases, high-traffic web servers, or latency-sensitive services.
• Security hardening:apply specific kernel-level security settings or restrict certain system behaviors to reduce the attack surface.
• Resource management:fine-tune how kubelet manages PIDs, disk space, image garbage collection, or CPU and memory resources.
• Workload compatibility:help ensure that the node environment meets specific prerequisites for specialized software or older applications that require particular kernel settings.

Other options for customizing node configurations

You can also customize your node configuration by using other methods:

• Runtime configuration file: to customize a containerd container runtime on your GKE nodes, you can use a different file called a runtime configuration file . For more information, see Customize containerd configuration in GKE nodes .
• ComputeClass: you can specify node attributes in your GKE ComputeClass specification. You can use ComputeClasses, in both GKE Autopilot mode and Standard mode, in GKE version 1.32.1-gke.1729000 and later. For more information, see Customize the node system configuration .
• DaemonSets: you can also use DaemonSets to customize nodes. For more information, see Automatically bootstrapping GKE nodes with DaemonSets .

Node system configurations are not supported in Windows Server nodes.

Before you begin

Before you begin, make sure to do the following:

• Install command-line tools:
  • If you use the gcloud CLI examples in this document, ensure that you install and configure the Google Cloud CLI .
  • If you use the Terraform examples, ensure that you install and configure Terraform .
• Grant permissions: you need appropriate IAM permissions to create and update GKE clusters and node pools, such as container.clusterAdmin or a different role with equivalent permissions.
• Plan for potential workload disruption: custom node configurations are applied at the node pool level. Changes typically trigger a rolling update of the nodes in the pool, which involves re-creating the nodes . Plan for potential workload disruption and use Pod Disruption Budgets (PDBs) where appropriate.
• Back up and test all changes:always test configuration changes in a staging or development environment before you apply them to production. Incorrect settings can lead to node instability or workload failures.
• Review GKE default settings:GKE node images come with optimized default configurations. Only customize parameters if you have a specific need and understand the impact of your changes.

Use a node system configuration in GKE Standard mode

When you use a node system configuration, you use a YAML file that contains the configuration parameters for the kubelet and the Linux kernel. Although node system configurations are also available in GKE Autopilot mode, the steps in this document show you how to create and use a configuration file for GKE Standard mode.

To use a node system configuration in GKE Standard mode, do the following:

1. Create a configuration file . This file contains your kubelet and sysctl configurations.
2. Add the configuration when you create a cluster, or when you create or update a node pool.

Create a configuration file

Write your node system configuration in YAML. The following example adds configurations for the kubelet and sysctl options:

  kubeletConfig 
 : 
  
 cpuManagerPolicy 
 : 
  
 static 
  
 allowedUnsafeSysctls 
 : 
  
 - 
  
 'kernel.shm*' 
  
 - 
  
 'kernel.msg*' 
  
 - 
  
 'kernel.sem' 
  
 - 
  
 'fs.mqueue*' 
  
 - 
  
 'net.*' 
 linuxConfig 
 : 
  
 sysctl 
 : 
  
 net.core.somaxconn 
 : 
  
 '2048' 
  
 net.ipv4.tcp_rmem 
 : 
  
 '4096 
  
 87380 
  
 6291456' 
 

In this example, the following applies:

• The cpuManagerPolicy: static field configures the kubelet to use the static CPU management policy . + The net.core.somaxconn: '2048' field limits the socket listen() backlog to 2,048 bytes.
• The net.ipv4.tcp_rmem: '4096 87380 6291456' field sets the minimum, default, and maximum value of the TCP socket receive buffer to 4,096 bytes, 87,380 bytes, and 6,291,456 bytes, respectively.

If you want to add configurations only for the kubelet or sysctl , include only that section in your node system configuration. For example, to add a kubelet configuration, create the following file:

  kubeletConfig 
 : 
  
 cpuManagerPolicy 
 : 
  
 static 
 

For a complete list of the fields that you can add to your node system configuration, see the Kubelet configuration options and Sysctl configuration options sections.

Add the configuration to a Standard node pool

After you create the node system configuration, add the --system-config-from-file flag by using the Google Cloud CLI. You can add this flag when you create a cluster, or when you create or update a node pool. You can't add a node system configuration by using the Google Cloud console.

Create a cluster with the node system configuration

You can add a node system configuration during cluster creation by using the gcloud CLI or Terraform. The following instructions apply the node system configuration to the default node pool:

 gcloud  
container  
clusters  
create  
 CLUSTER_NAME 
  
 \ 
  
--location = 
 LOCATION 
  
 \ 
  
--system-config-from-file = 
 SYSTEM_CONFIG_PATH 
 

 resource "google_container_cluster" "default" {
  name     = "gke-standard-regional-cluster"
  location = "us-central1"

  initial_node_count = 1

  node_config {
    # Kubelet configuration
    kubelet_config {
      cpu_manager_policy = "static"
    }

    linux_node_config {
      # Sysctl configuration
      sysctls = {
        "net.core.netdev_max_backlog" = "10000"
      }

      # Linux cgroup mode configuration
      cgroup_mode = "CGROUP_MODE_V2"

      # Linux huge page configuration
      hugepages_config {
        hugepage_size_2m = "1024"
      }
    }
  }
} 

Create a new node pool with the node system configuration

You can add a node system configuration when you use the gcloud CLI or Terraform to create a new node pool.

The following instructions apply the node system configuration to a new node pool:

 gcloud  
container  
node-pools  
create  
 POOL_NAME 
  
 \ 
  
--cluster  
 CLUSTER_NAME 
  
 \ 
  
--location = 
 LOCATION 
  
 \ 
  
--system-config-from-file = 
 SYSTEM_CONFIG_PATH 
 

 resource "google_container_node_pool" "default" {
  name    = "gke-standard-regional-node-pool"
  cluster = google_container_cluster.default.name

  node_config {
    # Kubelet configuration
    kubelet_config {
      cpu_manager_policy = "static"
    }

    linux_node_config {
      # Sysctl configuration
      sysctls = {
        "net.core.netdev_max_backlog" = "10000"
      }

      # Linux cgroup mode configuration
      cgroup_mode = "CGROUP_MODE_V2"

      # Linux huge page configuration
      hugepages_config {
        hugepage_size_2m = "1024"
      }
    }
  }
} 

Update the node system configuration of an existing node pool

You can update the node system configuration of an existing node pool by running the following command:

   
gcloud  
container  
node-pools  
update  
 POOL_NAME 
  
 \ 
  
--cluster = 
 CLUSTER_NAME 
  
 \ 
  
--location = 
 LOCATION 
  
 \ 
  
--system-config-from-file = 
 SYSTEM_CONFIG_PATH 
 

Replace the following:

• POOL_NAME : the name of the node pool that you want to update
• CLUSTER_NAME : the name of the cluster that you want to update
• LOCATION : the compute zone or region of the cluster
• SYSTEM_CONFIG_PATH : the path to the file that contains your kubelet and sysctl configurations

This change requires re-creating the nodes, which can cause disruption to your running workloads. For more information about this specific change, find the corresponding row in the manual changes that re-create the nodes using a node upgrade strategy without respecting maintenance policies table.

For more information about node updates, see Planning for node update disruptions .

Edit a node system configuration

To edit a node system configuration, you can create a new node pool with the configuration that you want, or update the node system configuration of an existing node pool.

Edit by creating a node pool

To edit a node system configuration by creating a node pool, do the following:

1. Create a configuration file with the configuration that you want.
2. Add the configuration to a new node pool.
3. Migrate your workloads to the new node pool.
4. Delete the old node pool .

Edit by updating an existing node pool

To edit the node system configuration of an existing node pool, follow the instructions in the Update node pool tab for adding the configuration to a node pool . When you update a node system configuration and the new configuration overrides the node pool's existing system configuration, the nodes must be re-created. If you omit any parameters during an update, the parameters are set to their respective defaults.

If you want to reset the node system configuration back to the defaults, update your configuration file with empty values for the kubelet and sysctl fields, for example:

  kubeletConfig 
 : 
  
 {} 
 linuxConfig 
 : 
  
 sysctl 
 : 
  
 {} 
 

Delete a node system configuration

To remove a node system configuration, do the following steps:

1. Create a node pool .
2. Migrate your workloads to the new node pool.
3. Delete the node pool that has the old node system configuration.

Configuration options for the kubelet

The tables in this section describe the kubelet options that you can modify.

CPU management

The following table describes the CPU management options for the kubelet .

Memory management and eviction

The following table describes the modifiable options for memory management and eviction. This section also contains a separate table that describes the modifiable options for the evictionSoft flag.

The following table shows the modifiable options for the evictionSoft flag. The same options also apply to the evictionSoftGracePeriod and evictionMinimumReclaim flags with different restrictions.

PID management

The following table describes the modifiable options for PID management.

Logging

The following table describes the modifiable options for logging.

Image garbage collection

The following table describes the modifiable options for image garbage collection.

Image pulling

The following table describes the modifiable options for image pulling.

Security and unsafe operations

The following table describes the modifiable options for configuring security and handling unsafe operations.

Resource Managers

Kubernetes offers a suite of Resource Managers. You can configure these Resource Managers to coordinate and optimize the alignment of node resources for Pods that are configured with specific requirements for CPUs, devices, and memory (hugepages) resources.

The following table describes the modifiable options for Resource Managers.

Sysctl configuration options

To tune the performance of your system, you can modify Linux kernel parameters. The tables in this section describe the various kernel parameters that you can configure.

Filesystem parameters ( fs.* )

The following table describes the modifiable parameters for the Linux filesystem. These settings control the behavior of the Linux filesystem, such as file handle limits and event monitoring.

Kernel parameters ( kernel.* )

The following table describes the modifiable parameters for the Linux kernel. These settings configure core kernel functionalities, including shared memory allocation.

Network parameters ( net.* )

The following table describes the modifiable parameters for networking. These settings tune the performance and behavior of the networking stack, from socket buffers to connection tracking.

Virtual Memory parameters ( vm.* )

The following table describes the modifiable parameters for the Virtual Memory subsystem. These settings manage the Virtual Memory subsystem, which controls how the kernel handles memory, swapping, and disk caching.

For more information about the supported values for each sysctl flag, see the --system-config-from-file gcloud CLI documentation .

Different Linux namespaces might have unique values for a given sysctl flag, but others might be global for the entire node. Updating sysctl options by using a node system configuration helps ensure that the sysctl is applied globally on the node and in each namespace, so that each Pod has identical sysctl values in each Linux namespace.

Linux cgroup mode configuration options

The container runtime and kubelet use Linux kernel cgroups for resource management, such as limiting how much CPU or memory each container in a Pod can access. There are two versions of the cgroup subsystem in the kernel: cgroupv1 and cgroupv2 . Kubernetes support for cgroupv2 was introduced as alpha in Kubernetes version 1.18, beta in 1.22, and GA in 1.25. For more information, see the Kubernetes cgroups v2 documentation .

Node system configuration lets you customize the cgroup configuration of your node pools. You can use cgroupv1 or cgroupv2 . GKE uses cgroupv2 for new Standard node pools that run version 1.26 and later, and cgroupv1 for node pools that run versions earlier than 1.26. For node pools that were created with node auto-provisioning, the cgroup configuration depends on the initial cluster version, not the node pool version. cgroupv1 is not supported on Arm machines.

You can use node system configuration to change the setting for a node pool to use cgroupv1 or cgroupv2 explicitly. Upgrading an existing node pool that uses cgroupv1 to version 1.26 doesn't change the setting to cgroupv2 . Existing node pools that run a version earlier than 1.26—and that don't include a customized cgroup configuration—will continue to use cgroupv1 . To change the setting, you must explicitly specify cgroupv2 for the existing node pool.

For example, to configure your node pool to use cgroupv2 , use a node system configuration file such as the following:

  linuxConfig 
 : 
  
 cgroupMode 
 : 
  
 'CGROUP_MODE_V2' 
 

The supported cgroupMode options are the following:

• CGROUP_MODE_V1 : use cgroupv1 on the node pool.
• CGROUP_MODE_V2 : use cgroupv2 on the node pool.
• CGROUP_MODE_UNSPECIFIED : use the default GKE cgroup configuration.

To use cgroupv2 , the following requirements and limitations apply:

• For a node pool that runs a version earlier than 1.26, you must use gcloud CLI version 408.0.0 or later. Alternatively, use gcloud beta with version 395.0.0 or later.
• Your cluster and node pools must run GKE version 1.24.2-gke.300 or later.
• You must use either the Container-Optimized OS with containerd or Ubuntu with containerd node image .
• If any of your workloads depend on reading the cgroup filesystem ( /sys/fs/cgroup/... ), ensure that they are compatible with the cgroupv2 API.
• If you use any monitoring or third-party tools, ensure that they are compatible with cgroupv2 .
• If you use Java workloads (JDK), we recommend that you use versions which fully support cgroupv2 , including JDK 8u372 , JDK 11.0.16 or later, or JDK 15 or later.

Verify cgroup configuration

When you add a node system configuration, GKE must re-create the nodes to implement the changes. After you add the configuration to a node pool and the nodes are re-created, you can verify the new configuration.

You can verify the cgroup configuration for nodes in a node pool by using gcloud CLI or the kubectl command-line tool:

 gcloud  
container  
node-pools  
describe  
 POOL_NAME 
  
 \ 
  
--format = 
 'value(Config.effectiveCgroupMode)' 
 

Linux hugepages configuration options

You can use a node system configuration file to pre-allocate hugepages . Kubernetes supports pre-allocated hugepages as a resource type, similar to CPU or memory.

To use hugepages, the following limitations and requirements apply:

• To ensure that the node is not fully occupied by hugepages, the overall size of the allocated hugepages can't exceed either the following:
  • On machines with less than 30 GB memory: 60% of the total memory. For example, on e2-standard-2 machine with 8 GB of memory, you can't allocate more than 4.8 GB for hugepages.
  • On machines with more than 30 GB memory: 80% of the total memory. For example, on c4a-standard-8 machines with 32 GB of memory, hugepages cannot exceed 25.6 GB.
• 1 GB hugepages are only available on A3, C2D, C3, C3D, C4, C4A, C4D, CT5E, CT5LP, CT6E, H3, M2, M3, M4, or Z3 machine types.

The following table describes the modifiable settings for Linux hugepages.

Transparent hugepages (THP)

You can use a node system configuration file to enable the Linux kernel's Transparent HugePage Support . With THP, the kernel automatically assigns hugepages to processes without manual pre-allocation.

The following table describes the modifiable parameters for THP.

THP is available on GKE version 1.33.2-gke.4655000 or later. It is also enabled on new TPU node pools by default on GKE version 1.33.2-gke.4655000 or later. THP isn't enabled when you upgrade existing node pools to a supported version or later.

What's next

• Learn more about node pools .
• Learn how to create node pools .