Version 1.16. This version is no longer supported. For information about how to upgrade to version 1.28, see Upgrade clusters in the latest documentation. For more information about supported and unsupported versions, see the Versioning page in the latest documentation.

Implement flat-mode network model with BGP support

This document describes how to implement a flat-mode network model with Border Gateway Protocol (BGP) support. When you implement a network model with BGP support, BGP dynamically ensures that pods in different Layer 2 domains can communicate with each other. Flat-mode networking with BGP is sometimes called dynamic flat IP.

For more information about flat-mode network models, see Flat vs island mode network models .

How to implement a flat-mode network that uses BGP

Flat-mode networking with BGP is enabled when you create a new cluster. You can't enable this feature for an existing cluster. Once this feature is enabled, you can make changes to some of the configuration settings.

To implement a cluster on a flat-mode network model with BGP support:

Edit the cluster configuration file:
- Set the spec.clusterNetwork.advancedNetworking field to true .
- If you want to enable flat-mode networking for IPv4, set the spec.clusterNetwork.flatIPv4 field to true .
  
  For an alternative, see Dual-stack cluster (IPv4 Island, IPv6 Dynamic Flat IP) , which configures your cluster with flat-mode networking for IPv6 only.
```
  apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1 
 kind 
 : 
  
 Cluster 
 metadata 
 : 
  
 name 
 : 
  
 bm 
  
 namespace 
 : 
  
 cluster-bm 
 spec 
 : 
  
 type 
 : 
  
 user 
  
 ... 
   
 clusterNetwork 
 : 
  
 advancedNetworking 
 : 
  
 true 
  
 flatIPv4 
 : 
  
 true 
  
 ... 
 
```
When spec.clusterNetwork.flatIPv4 is set to true , the field spec.clusterNetwork.pods.cidrBlocks is ignored and can be omitted. However, you need to add a ClusterCIDRConfigs manifest in the cluster configuration file (per-node, per-nodepool and/or per-cluster).
Append a NetworkGatewayGroup manifest to the cluster configuration file:

Specify the floating IPs to use for BGP peering. Ensure that the resource name is default and the namespace is the cluster namespace.
```
  --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 NetworkGatewayGroup 
 metadata 
 : 
  
 name 
 : 
  
 default 
  
 namespace 
 : 
  
 cluster-bm 
 spec 
 : 
  
 floatingIPs 
 : 
  
 - 
  
 10.0.1.100 
  
 - 
  
 10.0.2.100 
 
```
The NetworkGatewayGroup custom resource manages a list of one or more floating IP addresses. The BGP peering sessions are initiated from floating IP addresses that you specify in the NetworkGatewayGroup custom resource.
Append a FlatIPMode manifest to the cluster configuration file:

The name of the FlatIPMode resource must be default and the namespace is the cluster namespace. The peerSelector value flatip-peer: "true" matches the labels in BGPPeer objects bgppeer1 and bgppeer2 (defined in the following step), so both peers are used for flat-mode networking.

The following FlatIPMode manifest is for IPv4 single-stack, flat-mode networking with BGP. For alternative configurations, see Configuration examples .
```
  --- 
 apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1alpha1 
 kind 
 : 
  
 FlatIPMode 
 metadata 
 : 
  
 name 
 : 
  
 default 
  
 namespace 
 : 
  
 cluster-bm 
 spec 
 : 
  
 enableBGPIPv4 
 : 
  
 true 
  
 enableBGPIPv6 
 : 
  
 false 
  
 peerSelector 
 : 
  
 flatip-peer 
 : 
  
 "true" 
 
```

Append one or more BGPPeer manifests to the cluster configuration file:

You choose the names for the resources, but all BGPPeer resources must be in the cluster namespace.

  --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 BGPPeer 
 metadata 
 : 
  
 name 
 : 
  
 bgppeer1 
  
 namespace 
 : 
  
 cluster-bm 
  
 labels 
 : 
  
 flatip-peer 
 : 
  
 "true" 
 spec 
 : 
  
 localASN 
 : 
  
 65001 
  
 peerASN 
 : 
  
 65000 
  
 peerIP 
 : 
  
 10.0.1.254 
  
 sessions 
 : 
  
 2 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 BGPPeer 
 metadata 
 : 
  
 name 
 : 
  
 bgppeer2 
  
 namespace 
 : 
  
 cluster-bm 
  
 labels 
 : 
  
 flatip-peer 
 : 
  
 "true" 
 spec 
 : 
  
 localASN 
 : 
  
 65001 
  
 peerASN 
 : 
  
 65000 
  
 peerIP 
 : 
  
 10.0.2.254 
  
 sessions 
 : 
  
 2

Append a ClusterCIDRConfig manifest to the cluster configuration file:

The CusterCIDRConfig resource must also be in the cluster namespace.
```
  apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1alpha1 
 kind 
 : 
  
 ClusterCIDRConfig 
 metadata 
 : 
  
 name 
 : 
  
 cluster-wide-1 
  
 namespace 
 : 
  
 cluster-bm 
 spec 
 : 
  
 ipv4 
 : 
  
 cidr 
 : 
  
 "192.168.0.0/16" 
  
 perNodeMaskSize 
 : 
  
 24 
 
```
ClusterCIDRConfig is a custom resource that specifies Pod CIDR ranges to be allocated to nodes dynamically. The CNI uses the Pod CIDR ranges allocated on a Node to allocate IP addresses to the individual Pods running on the Node. The ClusterCIDRConfig is also used for dual-stack networking . For more information about the ClusterCIDRConfig custom resource, including usage examples, see Understand the ClusterCIDRConfig custom resource .
Create the cluster:
```
 bmctl  
create  
cluster 
```
For more information about creating clusters, see Cluster creation overview .

If your environment supports multi-protocol BGP (MP-BGP), IPv4 and IPv6 routes can be advertised over these IPv4 sessions. For examples of different configurations, including examples that use MP-BGP, see Configuration examples .

Modify your BGP-based flat-mode networking configuration

After you've created your cluster configured to use a flat-mode network model with BGP, some configuration settings can be updated. Use the admin cluster kubeconfig file when you make subsequent updates to the BGP-related resources ( NetworkGatewayGroup , FlatIPMode , and BGPPeer ). The admin cluster then reconciles the changes to the user cluster. If you edit these resources on the user cluster directly, the admin cluster overwrites your changes in subsequent reconciliations.

Example configurations

The following sections include cluster configuration examples for different variations of the flat-mode network model with BGP. The sample configuration files aren't complete. Most cluster settings that aren't relevant to flat-mode networking with BGP have been omitted.

Single-stack IPv4 cluster

The following cluster configuration file sample shows the settings for configuring a single-stack IPv4 cluster with flat-mode networking with BGP:

  apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1 
 kind 
 : 
  
 Cluster 
 metadata 
 : 
  
 name 
 : 
  
 bm 
  
 namespace 
 : 
  
 cluster-bm 
 spec 
 : 
  
 ... 
   
 clusterNetwork 
 : 
  
 advancedNetworking 
 : 
  
 true 
  
 flatIPv4 
 : 
  
 true 
  
 services 
 : 
  
 cidrBlocks 
 : 
  
 - 
  
 10.96.0.0/12 
  
 ... 
 --- 
 apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1alpha1 
 kind 
 : 
  
 ClusterCIDRConfig 
  
 metadata 
 : 
  
 name 
 : 
  
 cluster-wide-1 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
  
 ipv4 
 : 
  
 cidr 
 : 
  
 "222.2.0.0/16" 
  
 perNodeMaskSize 
 : 
  
 24 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 NetworkGatewayGroup 
 metadata 
 : 
  
 name 
 : 
  
 default 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
  
 floatingIPs 
 : 
  
 - 
  
 10.0.1.100 
  
 - 
  
 10.0.3.100 
 --- 
 apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1alpha1 
 kind 
 : 
  
 FlatIPMode 
 metadata 
 : 
  
 name 
 : 
  
 default 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
   
 enableBGPIPv4 
 : 
  
 true 
  
 enableBGPIPv6 
 : 
  
 false 
  
 peerSelector 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 BGPPeer 
 metadata 
 : 
  
 name 
 : 
  
 bgppeer1 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
  
 labels 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 spec 
 : 
  
 localASN 
 : 
  
 65001 
  
 peerASN 
 : 
  
 65002 
  
 peerIP 
 : 
  
 10.0.1.254 
  
 sessions 
 : 
  
 2 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 BGPPeer 
 metadata 
 : 
  
 name 
 : 
  
 bgppeer2 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
  
 labels 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 spec 
 : 
  
 localASN 
 : 
  
 65001 
  
 peerASN 
 : 
  
 65002 
  
 peerIP 
 : 
  
 10.0.3.254 
  
 sessions 
 : 
  
 2

Dual-stack cluster (IPv4 Island, IPv6 Dynamic Flat IP)

The following cluster configuration file sample shows the settings for configuring a dual-stack (IPv4/IPv6) cluster with flat-mode networking with BGP for just IPv6:

  apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1 
 kind 
 : 
  
 Cluster 
 metadata 
 : 
  
 name 
 : 
  
 bm 
  
 namespace 
 : 
  
 cluster-bm 
 spec 
 : 
  
 ... 
   
 clusterNetwork 
 : 
  
 advancedNetworking 
 : 
  
 true 
  
 flatIPv4 
 : 
  
 false 
  
 pods 
 : 
  
 cidrBlocks 
 : 
  
 - 
  
 192.168.0.0/16 
  
 services 
 : 
  
 cidrBlocks 
 : 
  
 - 
  
 10.96.0.0/12 
   
 # Additional IPv6 CIDR block determines if the cluster is dual-stack 
  
 - 
  
 2620:0:1000:2630:5:2::/112 
  
 ... 
  
 --- 
 apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1alpha1 
 kind 
 : 
  
 ClusterCIDRConfig 
  
 metadata 
 : 
  
 name 
 : 
  
 cluster-wide-1 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
  
 ipv4 
 : 
  
 cidr 
 : 
  
 "192.168.0.0/16" 
  
 perNodeMaskSize 
 : 
  
 24 
  
 ipv6 
 : 
  
 cidr 
 : 
  
 "2222:3::/112" 
  
 perNodeMaskSize 
 : 
  
 120 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 NetworkGatewayGroup 
 metadata 
 : 
  
 name 
 : 
  
 default 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
  
 floatingIPs 
 : 
  
 - 
  
 10.0.1.100 
  
 - 
  
 10.0.3.100 
 --- 
 apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1alpha1 
 kind 
 : 
  
 FlatIPMode 
 metadata 
 : 
  
 name 
 : 
  
 default 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
   
 enableBGPIPv4 
 : 
  
 false 
  
 enableBGPIPv6 
 : 
  
 true 
  
 peerSelector 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 BGPPeer 
 metadata 
 : 
  
 name 
 : 
  
 bgppeer1 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
  
 labels 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 spec 
 : 
  
 localASN 
 : 
  
 65001 
  
 peerASN 
 : 
  
 65002 
  
 peerIP 
 : 
  
 10.0.1.254 
  
 sessions 
 : 
  
 2 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 BGPPeer 
 metadata 
 : 
  
 name 
 : 
  
 bgppeer2 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
  
 labels 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 spec 
 : 
  
 localASN 
 : 
  
 65001 
  
 peerASN 
 : 
  
 65002 
  
 peerIP 
 : 
  
 10.0.3.254 
  
 sessions 
 : 
  
 2

Dual-stack cluster (IPv4 Dynamic Flat IP, IPv6 Dynamic Flat IP)

The following cluster configuration file sample shows the settings for configuring a dual-stack cluster with flat-mode networking with BGP:

  apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1 
 kind 
 : 
  
 Cluster 
 metadata 
 : 
  
 name 
 : 
  
 bm 
  
 namespace 
 : 
  
 cluster-bm 
 spec 
 : 
  
 ... 
   
 clusterNetwork 
 : 
  
 advancedNetworking 
 : 
  
 true 
  
 flatIPv4 
 : 
  
 true 
  
 pods 
 : 
  
 cidrBlocks 
 : 
  
 - 
  
 192.168.0.0/16 
  
 services 
 : 
  
 cidrBlocks 
 : 
  
 - 
  
 10.96.0.0/12 
   
 # Additional IPv6 CIDR block determines if the cluster is dual-stack 
  
 - 
  
 2620:0:1000:2630:5:2::/112 
  
 ... 
  
 --- 
 apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1alpha1 
 kind 
 : 
  
 ClusterCIDRConfig 
  
 metadata 
 : 
  
 name 
 : 
  
 cluster-wide-1 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
  
 ipv4 
 : 
  
 cidr 
 : 
  
 "222.2.0.0/16" 
  
 perNodeMaskSize 
 : 
  
 24 
  
 ipv6 
 : 
  
 cidr 
 : 
  
 "2222:3::/112" 
  
 perNodeMaskSize 
 : 
  
 120 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 NetworkGatewayGroup 
 metadata 
 : 
  
 name 
 : 
  
 default 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
  
 floatingIPs 
 : 
  
 - 
  
 10.0.1.100 
  
 - 
  
 10.0.3.100 
 --- 
 apiVersion 
 : 
  
 baremetal.cluster.gke.io/v1alpha1 
 kind 
 : 
  
 FlatIPMode 
 metadata 
 : 
  
 name 
 : 
  
 default 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
 spec 
 : 
   
 enableBGPIPv4 
 : 
  
 true 
  
 enableBGPIPv6 
 : 
  
 true 
  
 peerSelector 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 BGPPeer 
 metadata 
 : 
  
 name 
 : 
  
 bgppeer1 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
  
 labels 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 spec 
 : 
  
 localASN 
 : 
  
 65001 
  
 peerASN 
 : 
  
 65002 
  
 peerIP 
 : 
  
 10.0.1.254 
  
 sessions 
 : 
  
 2 
 --- 
 apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 BGPPeer 
 metadata 
 : 
  
 name 
 : 
  
 bgppeer2 
  
 namespace 
 : 
  
 cluster-bm 
  
 # Must match the cluster namespace 
  
 labels 
 : 
  
 flatipmode-peer 
 : 
  
 "true" 
 spec 
 : 
  
 localASN 
 : 
  
 65001 
  
 peerASN 
 : 
  
 65002 
  
 peerIP 
 : 
  
 10.0.3.254 
  
 sessions 
 : 
  
 2

Troubleshooting

To help you troubleshoot issues related to flat-mode networking with BGP, this section includes instructions for checking your configuration:

Verify if a FlatIPModes object is created in the cluster namespace on the admin cluster:

 kubectl  
get  
flatipmodes  
-A  
--kubeconfig  
 ADMIN_KUBECONFIG

The response should look something like this:

 NAMESPACE                 NAME      AGE
cluster-bm                default   2d17h

Verify if a flatipmodes.networking.gke.io object is created on the user cluster:

The flatipmodes.networking.gke.io object is cluster scoped.
```
 kubectl  
get  
flatipmodes.networking.gke.io  
--kubeconfig  
 USER_KUBECONFIG 
 
```
The response should look something like this:
```
 NAME      AGE
default   2d17h 
```

Get the BGPSessions resources to view the current sessions:

 kubectl  
get  
bgpsessions  
-A  
--kubeconfig  
 USER_KUBECONFIG