Using GKE Dataplane V2

This page explains how to enable and troubleshoot GKE Dataplane V2 for Google Kubernetes Engine (GKE) clusters.

New Autopilot clusters have GKE Dataplane V2 enabled in versions 1.22.7-gke.1500 and later and versions 1.23.4-gke.1500 and later. If you're experiencing issues with using GKE Dataplane V2, skip to Troubleshooting .

Creating a GKE cluster with GKE Dataplane V2

You can enable GKE Dataplane V2 when you create new clusters with GKE version 1.20.6-gke.700 and later by using the gcloud CLI or the Kubernetes Engine API. You can also enable GKE Dataplane V2 in Preview when you create new clusters with GKE version 1.17.9 and later

 gcloud  
container  
clusters  
create  
 CLUSTER_NAME 
  
 \ 
  
--enable-dataplane-v2  
 \ 
  
--enable-ip-alias  
 \ 
  
--release-channel  
 CHANNEL_NAME 
  
 \ 
  
--location  
 COMPUTE_LOCATION 
 

  "cluster" 
 :{ 
  
 "initialClusterVersion" 
 : 
 " VERSION 
" 
 , 
  
 "ipAllocationPolicy" 
 :{ 
  
 "useIpAliases" 
 : 
 true 
  
 }, 
  
 "networkConfig" 
 :{ 
  
 "datapathProvider" 
 : 
 "ADVANCED_DATAPATH" 
  
 }, 
  
 "releaseChannel" 
 :{ 
  
 "channel" 
 : 
 " CHANNEL_NAME 
" 
  
 } 
 } 
 

Troubleshooting issues with GKE Dataplane V2

This section shows you how to investigate and resolve issues with GKE Dataplane V2.

1. Confirm that GKE Dataplane V2 is enabled:

    kubectl  
   -n  
   kube-system  
   get  
   pods  
   -l  
   k8s-app = 
   cilium  
   -o  
   wide 
   

   If GKE Dataplane V2 is running, the output includes Pods with the prefix anetd- . anetd is the networking controller for GKE Dataplane V2.

2. If the issue is with services or network policy enforcement, check the anetd Pod logs. Use the following log selectors in Cloud Logging:

    resource.type = 
    "k8s_container" 
   labels. "k8s-pod/k8s-app" 
    = 
    "cilium" 
   resource.labels.cluster_name = 
    " CLUSTER_NAME 
   " 
    
   

3. If Pod creation is failing, check the kubelet logs for clues. Use the following log selectors in Cloud Logging:

    resource.type = 
    "k8s_node" 
    log_name 
    = 
   ~ ".*/logs/kubelet" 
   resource.labels.cluster_name = 
    " CLUSTER_NAME 
   " 
    
   

   Replace CLUSTER_NAME with the name of the cluster, or remove it entirely to see logs for all clusters.

Known issues

Intermittent connectivity issues related to NodePort range conflicts in GKE Dataplane V2 cluster

In GKE Dataplane V2 clusters, intermittent connectivity problems can occur for masqueraded traffic or with ephemeral port usage. These problems are due to the potential port conflicts with the reserved NodePort range and typically happen in the following scenarios:

• Custom ip-masq-agent :If you are using a custom ip-masq-agent (version 2.10 or later), where the cluster has NodePort or Load Balancer services, you might observe intermittent connectivity issues due to their conflict with the NodePort range. Since version 2.10 and later, ip-masq-agent has the --random-fully argument implemented internally by default. To mitigate this, explicitly set --random-fully=false (applicable since version 2.11) under arguments in your ip-masq-agent configuration. For configuration details, see Configuring an IP masquerade agent in Standard clusters .

• Ephemeral Port Range Overlap:If the ephemeral port range defined by net.ipv4.ip_local_port_range on your GKE nodes overlaps with the NodePort range (30000-32767), it can also trigger connectivity issues. To prevent this problem, ensure that these two ranges don't overlap.

Review your ip-masq-agent configuration and ephemeral port range settings to ensure they don't conflict with the NodePort range. If you encounter intermittent connectivity issues, consider these potential causes and adjust your configuration accordingly.

Network Policy port ranges don't take effect

If you specify an endPort field in a Network Policy on a cluster that has GKE Dataplane V2 enabled, it won't take effect.

Starting in GKE 1.22, the Kubernetes Network Policy API lets you specify a range of ports where the Network Policy is enforced. This API is supported in clusters with Calico Network Policy but is not supported in clusters with GKE Dataplane V2.

You can verify the behavior of your NetworkPolicy objects by reading them back after writing them to the API server. If the object still contains the endPort field, the feature is enforced. If the endPort field is missing, the feature is not enforced. In all cases, the object stored in the API server is the source of truth for the Network Policy.

For more information see KEP-2079: Network Policy to support Port Ranges .

Pods display failed to allocate for range 0: no IP addresses available in range set error message

Affected GKE versions: 1.22 to 1.25

GKE clusters running node pools that use containerd and have GKE Dataplane V2 enabled might experience IP address leak issues and exhaust all the Pod IP addresses on a node. A Pod scheduled on an affected node displays an error message similar to the following:

 failed to allocate for range 0: no IP addresses available in range set: 10.48.131.1-10.48.131.62 

For more information about the issue, see containerd issue #5768 .

Fixed versions

To fix this issue, upgrade your cluster to one of the following GKE versions:

• 1.22.17-gke.3100 or later.
• 1.23.16-gke.200 or later.
• 1.24.9-gke.3200 or later.
• 1.25.6-gke.200 or later.

Workarounds for standard GKE clusters

You can mitigate this issue by deleting the leaked Pod IP addresses for the node.

To delete the leaked Pod IP addresses, get authentication credentials for the cluster and run the following steps to clean up a single node, if you know its name.

1. Save the following shell script to a file named cleanup.sh :

     for 
     
    hash 
     
    in 
     
    $( 
   sudo  
   find  
   /var/lib/cni/networks/gke-pod-network  
   -iregex  
    '/var/lib/cni/networks/gke-pod-network/[0-9].*' 
     
   -exec  
   head  
   -n1  
    {} 
     
    \; 
    ) 
    ; 
     
    do 
     
    hash 
    = 
    " 
    ${ 
    hash 
    %%[[: 
    space 
    :]] 
    } 
    " 
    ; 
     
    if 
     
    [ 
     
   -z  
    $( 
   sudo  
   ctr  
   -n  
   k8s.io  
   c  
   ls  
    | 
     
   grep  
    $hash 
     
    | 
     
   awk  
    '{print $1}' 
    ) 
     
    ] 
    ; 
     
    then 
     
   sudo  
   grep  
   -ilr  
    $hash 
     
   /var/lib/cni/networks/gke-pod-network ; 
     
    fi 
    ; 
     
    done 
     
    | 
     
   sudo  
   xargs  
   -r  
   rm 
   

2. Run the script on a cluster node:

    gcloud  
   compute  
   ssh  
   --zone  
    "ZONE" 
     
   --project  
    "PROJECT" 
     
    NODE_NAME 
     
   --command  
    " 
    $( 
   cat  
   cleanup.sh ) 
    " 
    
   

   Replace NODE_NAME with the name of the node.

You can also run a DaemonSet version of this script to run in parallel on all nodes at once:

1. Save the following manifest to a file named cleanup-ips.yaml :

     apiVersion 
    : 
     
    apps/v1 
    kind 
    : 
     
    DaemonSet 
    metadata 
    : 
     
    name 
    : 
     
    cleanup-ipam-dir 
     
    namespace 
    : 
     
    kube-system 
    spec 
    : 
     
    selector 
    : 
     
    matchLabels 
    : 
     
    name 
    : 
     
    cleanup-ipam 
     
    template 
    : 
     
    metadata 
    : 
     
    labels 
    : 
     
    name 
    : 
     
    cleanup-ipam 
     
    spec 
    : 
     
    hostNetwork 
    : 
     
    true 
     
    securityContext 
    : 
     
    runAsUser 
    : 
     
    0 
     
    runAsGroup 
    : 
     
    0 
     
    containers 
    : 
     
    - 
     
    name 
    : 
     
    cleanup-ipam 
     
    image 
    : 
     
    gcr.io/gke-networking-test-images/ubuntu-test:2022 
     
    command 
    : 
     
    - 
     
    /bin/bash 
     
    - 
     
    -c 
     
    - 
     
    | 
     
    while true; do 
     
    for hash in $(find /hostipam -iregex '/hostipam/[0-9].*' -mmin +10 -exec head -n1 {} \; ); do 
     
    hash="${hash%%[[:space:]]}" 
     
    if [ -z $(ctr -n k8s.io c ls | grep $hash | awk '{print $1}') ]; then 
     
    grep -ilr $hash /hostipam 
     
    fi 
     
    done | xargs -r rm 
     
    echo "Done cleaning up /var/lib/cni/networks/gke-pod-network at $(date)" 
     
    sleep 120s 
     
    done 
     
    volumeMounts 
    : 
     
    - 
     
    name 
    : 
     
    host-ipam 
     
    mountPath 
    : 
     
    /hostipam 
     
    - 
     
    name 
    : 
     
    host-ctr 
     
    mountPath 
    : 
     
    /run/containerd 
     
    volumes 
    : 
     
    - 
     
    name 
    : 
     
    host-ipam 
     
    hostPath 
    : 
     
    path 
    : 
     
    /var/lib/cni/networks/gke-pod-network 
     
    - 
     
    name 
    : 
     
    host-ctr 
     
    hostPath 
    : 
     
    path 
    : 
     
    /run/containerd 
    
   

2. Run the daemonset on the cluster:

    kubectl  
   apply  
   -f  
   cleanup-ips.yaml 
   

   You must have kubectl access as an administrator of the cluster to run this command.

3. Check the logs of the running DaemonSet:

    kubectl  
   -n  
   kube-system  
   logs  
   -l  
    name 
    = 
   cleanup-ipam 
   

Network Policy drops a connection due to incorrect connection tracking lookup

When a client Pod connects to itself using a Service or the virtual IP address of an internal passthrough Network Load Balancer, the reply packet is not identified as a part of an existing connection due to incorrect conntrack lookup in the dataplane. This means that a Network Policy that restricts ingress traffic for the Pod is incorrectly enforced on the packet.

The impact of this issue depends on the number of configured Pods for the Service. For example, if the Service has 1 backend Pod, the connection always fails. If the Service has 2 backend Pods, the connection fails 50% of the time.

Fixed versions

To fix this issue, upgrade your cluster to one of the following GKE versions:

• 1.28.3-gke.1090000 or later.

Workarounds

You can mitigate this issue by configuring the port and containerPort in the Service manifest to be the same value.

Packet drops for hairpin connection flows

When a Pod creates a TCP connection to itself using a Service, such that the Pod is both the source and destination of the connection, GKE Dataplane V2 eBPF connection tracking incorrectly tracks the connection states, leading to leaked conntrack entries.

When a connection tuple (protocol, source/destination IP, and source/destination port) has been leaked, new connections using the same connection tuple might result in return packets being dropped.

Fixed versions

To fix this issue, upgrade your cluster to one of the following GKE versions:

• 1.28.3-gke.1090000 or later
• 1.27.11-gke.1097000 or later

Workarounds

Use one of the following workarounds:

• Enable TCP reuse (keep-alives) for applications running in Pods that might communicate with itself using a Service. This prevents the TCP FIN flag from being issued and avoid leaking the conntrack entry.

• When using short-lived connections, expose the Pod using a proxy load balancer, such as Gateway , to expose the Service. This results in the destination of the connection request being set to the load balancer IP address, preventing GKE Dataplane V2 from performing SNAT to the loopback IP address.

What's next

• Use network policy logging to record when connections to Pods are allowed or denied by your cluster's network policies .
• Learn how GKE Dataplane V2 works.