Configuring an IP masquerade agent in Standard clusters

This page explains how to configure clusters created in Google Kubernetes Engine (GKE) Standard mode to perform IP masquerade with the ip-masq-agent . For more information about IP masquerading in GKE Autopilot mode , see Use Egress NAT Policy to configure IP masquerade in Autopilot clusters .

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update .

Checking ip-masq-agent status

This section shows you how to:

• Determine whether your cluster has an ip-masq-agent DaemonSet.
• Check the ip-masq-agent ConfigMap resource.

Check the ip-masq-agent DaemonSet

To check if your cluster is running the ip-masq-agent DaemonSet, use the Google Cloud CLI or the Google Cloud console.

To create the ip-masq-agent ConfigMap and deploy the ip-masq-agent DaemonSet, refer to Configuring and deploying the ip-masq-agent .

Checking the ip-masq-agent ConfigMap

To check if your cluster is running the ip-masq-agent ConfigMap, use the Google Cloud CLI or the Google Cloud console.

If the cluster already has the ip-masq-agent ConfigMap, you can configure and deploy it.

Configuring and deploying the ip-masq-agent

This section shows you how to create or edit the ip-masq-agent ConfigMap and how to deploy or delete the ip-masq-agent DaemonSet. To determine which tasks you need to perform, you must first determine whether your cluster already has the ip-masq-agent ConfigMap and ip-masq-agent DaemonSet .

Creating the ip-masq-agent ConfigMap

The following steps show how to create the ip-masq-agent ConfigMap. If your cluster already has the ip-masq-agent ConfigMap, edit an existing ip-masq-agent ConfigMap instead .

1. Create a configuration file using the following template and save it locally. You can use any name for the local copy of this configuration file.

    nonMasqueradeCIDRs:  
   -  
    CIDR_1 
     
   -  
    CIDR_2 
   masqLinkLocal:  
    false 
   resyncInterval:  
    SYNC_INTERVAL 
    UNIT_OF_TIME 
    
   

   Replace the following:

   • CIDR_1 and CIDR_2 : the IP address ranges in CIDR format . When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. If you need more than two CIDRs, add more entries to the nonMasqueradeCIDRs list following the same format. At minimum, the nonMasqueradeCIDRs property should include the node and Pod IP address ranges of your cluster.

   • SYNC_INTERVAL : the amount of time after which each ip-masq-agent Pod checks the contents of the ip-masq-agent ConfigMap and writes any changes to its local /etc/config/ip-masq-agent file. Defaults to 60 .

   • UNIT_OF_TIME : the unit of time for the resyncInterval. Valid values include s (for seconds) or ms (for milliseconds). Defaults to s .

   Set masqLinkLocal to false (the default) unless you need to enable masquerading for packets sent to link local IPv4 addresses. For more information, see Masquerading to link-local destinations .

2. Create the ConfigMap resource:

    kubectl  
   create  
   configmap  
   ip-masq-agent  
    \ 
     
   --namespace = 
   kube-system  
    \ 
     
   --from-file = 
    config 
    = 
    LOCAL_CONFIG_FILE_PATH 
    
   

   Replace LOCAL_CONFIG_FILE_PATH with the path to the config file you created in the previous step.

3. Describe the ip-masq-agent ConfigMap in the kube-system namespace:

    kubectl  
   describe  
   configmaps/ip-masq-agent  
   -n  
   kube-system 
   

   The output is similar to the following:

    Name:         ip-masq-agent
   Namespace:    kube-system
   Labels:       <none>
   Annotations:  <none>
   
   Data
   ====
   config:
   ----
   nonMasqueradeCIDRs:
     - 198.15.5.92/24
     - 10.0.0.0/8
   masqLinkLocal: false
   resyncInterval: 60s
   
   BinaryData
   ====
   Events:  <none> 
   

   This output includes the config parameter with your configuration changes. You can now deploy the ip-masq-agent DeamonSet .

Editing an existing ip-masq-agent ConfigMap

You can modify the contents of an existing ip-masq-agent ConfigMap by completing the following steps:

1. Open the ConfigMap in a text editor:

    kubectl  
   edit  
   configmap  
   ip-masq-agent  
   --namespace = 
   kube-system 
   

2. Edit the content of the ConfigMap file:

     apiVersion 
    : 
     
    v1 
    data 
    : 
     
    config 
    : 
     
    | 
     
    nonMasqueradeCIDRs: 
     
    - CIDR_1 
    
     
    - CIDR_2 
    
     
    masqLinkLocal: false 
     
    resyncInterval: SYNC_INTERVAL 
    UNIT_OF_TIME 
    
    kind 
    : 
     
    ConfigMap 
    metadata 
    : 
     
    name 
    : 
     
    ip-masq-agent 
     
    namespace 
    : 
     
    kube-system 
    
   

   Replace the following:

   • CIDR_1 and CIDR_2 : the IP address ranges in CIDR format . When packets are sent to these destinations, your cluster does not masquerade IP address sources and preserves source Pod IP addresses. If you need more than two CIDRs, add more entries to the nonMasqueradeCIDRs list following the same format. At minimum, the nonMasqueradeCIDRs property should include the node and Pod IP address ranges of your cluster.

   • SYNC_INTERVAL : the amount of time after which each ip-masq-agent Pod checks the contents of the ip-masq-agent ConfigMap and writes any changes to its local /etc/config/ip-masq-agent file. Defaults to 60 .

   • UNIT_OF_TIME : the unit of time for the resyncInterval. Valid values include s (for seconds) or ms (for milliseconds). Defaults to s .

   Set masqLinkLocal to false (the default) unless you need to enable masquerading for packets sent to link local IPv4 addresses. For more information, see Masquerading to link-local destinations .

3. Describe the ip-masq-agent ConfigMap in the kube-system namespace:

    kubectl  
   describe  
   configmaps/ip-masq-agent  
   -n  
   kube-system 
   

   The output is similar to the following:

    Name:         ip-masq-agent
   Namespace:    kube-system
   Labels:       <none>
   Annotations:  <none>
   
   Data
   ====
   config:
   ----
   nonMasqueradeCIDRs:
     - 198.15.5.92/24
     - 10.0.0.0/8
   masqLinkLocal: false
   resyncInterval: 60s
   
   BinaryData
   ====
   
   Events:  <none> 
   

   This output includes the config parameter which matches the configuration value from the file you created.

Deploying the ip-masq-agent DaemonSet

After you create or edit your ip-masq-agent ConfigMap, deploy the ip-masq-agent DaemonSet.

1. Save the following manifest as a YAML file:

    apiVersion 
    : 
     
    apps/v1 
    kind 
    : 
     
    DaemonSet 
    metadata 
    : 
     
    name 
    : 
     
    ip-masq-agent 
     
    namespace 
    : 
     
    kube-system 
    spec 
    : 
     
    selector 
    : 
     
    matchLabels 
    : 
     
    k8s-app 
    : 
     
    ip-masq-agent 
     
    template 
    : 
     
    metadata 
    : 
     
    labels 
    : 
     
    k8s-app 
    : 
     
    ip-masq-agent 
     
    spec 
    : 
     
    hostNetwork 
    : 
     
    true 
     
    containers 
    : 
     
    - 
     
    name 
    : 
     
    ip-masq-agent 
     
    image 
    : 
     
    gke.gcr.io/ip-masq-agent:v2.12.3-gke.4@sha256:b5db41ddaf863b660da330322714f668101482b528829c50c53229d901d11af5 
     
    args 
    : 
     
    - 
     
    --v=2 
     
    - 
     
    --logtostderr=false 
     
    - 
     
    --log_file=/dev/stdout 
     
    - 
     
    --log_file_max_size=0 
     
    # The masq-chain must be IP-MASQ 
     
    - 
     
    --masq-chain=IP-MASQ 
     
    # To non-masquerade reserved IP ranges by default, 
     
    # uncomment the following line. 
     
    # - --nomasq-all-reserved-ranges 
     
    # Must be set to false when using Dataplane V2. 
     
    - 
     
    --random-fully=false 
     
    securityContext 
    : 
     
    privileged 
    : 
     
    false 
     
    capabilities 
    : 
     
    drop 
    : 
     
    [ 
    "ALL" 
    ] 
     
    add 
    : 
     
    [ 
    "NET_ADMIN" 
    , 
     
    "NET_RAW" 
    ] 
     
    allowPrivilegeEscalation 
    : 
     
    false 
     
    seccompProfile 
    : 
     
    type 
    : 
     
    RuntimeDefault 
     
    volumeMounts 
    : 
     
    - 
     
    name 
    : 
     
    config-volume 
     
    mountPath 
    : 
     
    /etc/config 
     
    volumes 
    : 
     
    - 
     
    name 
    : 
     
    config-volume 
     
    configMap 
    : 
     
    name 
    : 
     
    ip-masq-agent 
     
    optional 
    : 
     
    true 
     
    items 
    : 
     
    - 
     
    key 
    : 
     
    config 
     
    path 
    : 
     
    ip-masq-agent 
     
    tolerations 
    : 
     
    - 
     
    effect 
    : 
     
    NoSchedule 
     
    operator 
    : 
     
    Exists 
     
    - 
     
    effect 
    : 
     
    NoExecute 
     
    operator 
    : 
     
    Exists 
     
    - 
     
    key 
    : 
     
    "CriticalAddonsOnly" 
     
    operator 
    : 
     
    "Exists" 
   

   This manifest creates a volume named config-volume which is mounted as specified by the container's volumeMount.

   If you need to edit this manifest, consider the following conditions:

   • The volume name can be anything but must match the container's volumeMount name.

   • The ConfigMap name must match the name of the configMap referenced in the config-volume Volume in the Pod.

   • The name of the chain ( --masq-chain ) must be IP-MASQ . Otherwise, GKE does not override the default masquerading rules.

   • DaemonSet Pods read from the ip-masq-agent file. The ip-masq-agent file content is the value of the config key in the ConfigMap.

   • If you use non-masquerade reserved IP ranges by default, uncomment the - --nomasq-all-reserved-ranges line in the arg section.

2. Deploy the DaemonSet:

    kubectl  
   apply  
   -f  
    LOCAL_FILE_PATH 
    
   

   Replace LOCAL_FILE_PATH with the path to the file you created in the previous step.

You can manually update the ip-masq-agent DaemonSet you created. To learn more, see Updating a DaemonSet .

Deleting the ip-masq-agent

This section shows you how to delete the ip-masq-agent DaemonSet and the ip-masq-agent ConfigMap. Deleting the ip-masq-agent doesn't revert the existing IP masquerading settings on the nodes.

Deleting the ip-masq-agent DaemonSet

If you manually created the ip-masq-agent DaemonSet, you can delete it by running the following command:

 kubectl  
delete  
daemonsets  
ip-masq-agent  
-n  
kube-system 

Deleting the ip-masq-agent ConfigMap

To completely delete the ip-masq-agent ConfigMap, run the following command:

 kubectl  
delete  
configmap  
ip-masq-agent  
-n  
kube-system 

Troubleshooting

The following sections provide troubleshooting advice.

General troubleshooting

The following steps help you diagnose issues with IP address masquerading:

• Confirm the status of the ip-masq-agent . If the ConfigMap is not defined, traffic to all default destinations is not masqueraded and preserves the Pod IP address. Traffic to other destinations preserves the node IP address.
• Confirm the version of the ip-masq-agent image specified in the DaemonSet . If the version of the ip-masq-agent DaemonSet is not the latest, follow the deployment steps to update the DaemonSet .
• Check whether the IP-MASQ chain is correctly populated in the NAT IP tables by running the sudo iptables -t nat -L IP-MASQ command in the impacted node. If the nonMasqueradeCIDRs defined in the ConfigMap is not showing up on the NAT IP Tables, verify that there are no typos in the config file that was used to create the ConfigMap.
• Confirm that the destination allows both the node and Pod IP address ranges.
• If traffic is not accessible from the node or Pod, run a connectivity test .

Issue: Pod IP address changes to node IP address

While using an IP masquerade agent, you might notice the Pod's source IP address unexpectedly use the node's IP address when your Pods communicate with external destinations.

The issue is caused by a missing or incomplete custom Source Network Address Translation (SNAT) list. When you use an IP masquerade agent, the default SNAT of the cluster is used when the ConfigMap is missing or it doesn't contain a nonMasqueradeCIDRs list. When a packet leaves a Pod, the default SNAT changes the source IP address from the Pod's IP address to the node's internal IP address. To learn more about SNAT, see IP masquerade agent .

To resolve the issue, configure a custom SNAT list by defining a nonMasqueradeCIDRs list in the ip-masq-agent ConfigMap:

1. Open the ip-masq-agent ConfigMap:

    kubectl  
   edit  
   configmap  
   ip-masq-agent  
   --namespace = 
   kube-system 
   

2. Review the nonMasqueradeCIDRs list within the ConfigMap. The nonMasqueradeCIDRs list should be present and complete. For example:

     ... 
    nonMasqueradeCIDRs 
    : 
     
    - 
     
    35.100.0.0/16 
    ... 
    
   

3. If the list is missing or incomplete, add or modify the nonMasqueradeCIDRs list to include the destination IP ranges for which you want to preserve source Pod IP addresses. This list should also include the addresses that you want to use Cloud NAT.

   If you don't want any external traffic to use SNAT, set the nonMasqueradeCIDRs field to 0.0.0.0/0 . For example:

     ... 
    nonMasqueradeCIDRs 
    : 
     
    - 
     
    0.0.0.0/0 
    ... 
    
   

   When your traffic doesn't use SNAT, all packets sent from Pods retain the Pod's IP address as their source IP address, regardless of destination.

4. Check the source IP address of outbound packets from your Pods. To check this address, take a packet capture. If that's not feasible, you can check the IP address of the node, which should be the source IP address of outbound packets subject to SNAT, with the following command:

    kubectl  
   get  
   nodes  
   -o  
   wide 
   

What's next

• Learn about Alias IPs .
• Read the GKE network overview .
• Learn about configuring authorized networks .