Deploying multi-cluster Gateways

Autopilot Standard

This page describes how to deploy Kubernetes Gateway resources for load balancing ingress traffic across multiple Google Kubernetes Engine (GKE) clusters (or fleet). Before deploying multi-cluster Gateways, see Enabling multi-cluster Gateways to prepare your environment.

For deploying Gateways to load balance ingress traffic to just a single GKE cluster, see Deploying Gateways .

Multi-cluster Gateways

A multi-cluster Gateway is a Gateway resource that load balances traffic across multiple Kubernetes clusters. In GKE the gke-l7-global-external-managed-mc , gke-l7-regional-external-managed-mc , gke-l7-rilb-mc , and gke-l7-gxlb-mc GatewayClasses deploy multi-cluster Gateways that provide HTTP routing, traffic splitting, traffic mirroring, health-based failover, and more across different GKE clusters, Kubernetes Namespaces, and across different regions. Multi-cluster Gateways make managing application networking across many clusters and teams easy, secure, and scalable for infrastructure administrators.

A multi-cluster Gateway is a Gateway resource that load balances traffic
across multiple Kubernetes clusters.

This page introduces three examples to teach you how to deploy multi-cluster Gateways using the GKE Gateway controller :

Example 1 : An external, multi-cluster Gateway providing load balancing across two GKE clusters for internet traffic.
Example 2 : A private layer 7 cross regional Gateway.
Example 3 : Blue-green, weight-based traffic splitting and traffic mirroring across two GKE clusters for internal VPC traffic.
Example 4 : A capacity-based Gateway to load-balance request to different backends based on their max capacity.

Each of the examples will use the same store and site applications to model a real-world scenario where an online shopping service and a website service are owned and operated by separate teams and deployed across a fleet of shared GKE clusters. Each of the examples highlights different topologies and use cases enabled by multi-cluster Gateways.

Multi-cluster Gateways require some environmental preparation before they can be deployed. Before proceeding, follow the steps in Enabling multi-cluster Gateways :

Deploy GKE clusters.
Register your clusters to a fleet.
Enable the multi-cluster Service and multi-cluster Gateway controllers.

Lastly, review the GKE Gateway controller limitations and known issues before using it in your environment.

Multi-cluster, multi-region, external Gateway

In this tutorial, you will create an external multi-cluster Gateway that serves external traffic across an application running in two GKE clusters.

store.example.com is deployed across two GKE clusters and exposed
to the internet using a multi-cluster Gateway

In the following steps you:

Deploy the sample store application to the gke-west-1 and gke-east-1 clusters.
Configure Services on each cluster to be exported into your fleet (multi-cluster Services).
Deploy an external multi-cluster Gateway and an HTTPRoute to your config cluster ( gke-west-1 ).

After the application and Gateway resources are deployed, you can control traffic across the two GKE clusters using path-based routing:

Requests to /west are routed to store Pods in the gke-west-1 cluster.
Requests to /east are routed to store Pods in the gke-east-1 cluster.
Requests to any other path are routed to either cluster, according to its health, capacity, and proximity to the requesting client.

Deploying the demo application

Create the store Deployment and Namespace in all three of the clusters that were deployed in Enabling multi-cluster Gateways :

 kubectl  
apply  
--context  
gke-west-1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml
kubectl  
apply  
--context  
gke-west-2  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml
kubectl  
apply  
--context  
gke-east-1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml

It deploys the following resources to each cluster:

 namespace/store created
deployment.apps/store created

All examples in this page use the app deployed in this step. Make sure that the app is deployed across all three clusters before trying any of the remaining steps. This example uses only clusters gke-west-1 and gke-east-1 , and gke-west-2 is used in another example.

Multi-cluster Services

Services are how Pods are exposed to clients. Because the GKE Gateway controller uses container-native load balancing, it does not use the ClusterIP or Kubernetes load balancing to reach Pods. Traffic is sent directly from the load balancer to the Pod IP addresses. However, Services still play a critical role as a logical identifier for Pod grouping.

Multi-cluster Services (MCS) is an API standard for Services that span clusters and its GKE controller provides service discovery across GKE clusters. The multi-cluster Gateway controller uses MCS API resources to group Pods into a Service that is addressable across or spans multiple clusters.

The multi-cluster Services API defines the following custom resources:

ServiceExportsmap to a Kubernetes Service, exporting the endpoints of that Service to all clusters registered to the fleet. When a Service has a corresponding ServiceExport it means that the Service can be addressed by a multi-cluster Gateway.
ServiceImportsare automatically generated by the multi-cluster Service controller. ServiceExport and ServiceImport come in pairs. If a ServiceExport exists in the fleet, then a corresponding ServiceImport is created to allow the Service mapped to the ServiceExport to be accessed from across clusters.

Exporting Services works in the following way. A store Service exists in gke-west-1 which selects a group of Pods in that cluster. A ServiceExport is created in the cluster which allows the Pods in gke-west-1 to become accessible from the other clusters in the fleet. The ServiceExport will map to and expose Services that have the same name and Namespace as the ServiceExport resource.

  apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Service 
  metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 selector 
 : 
  
 app 
 : 
  
 store 
  
 ports 
 : 
  
 - 
  
 port 
 : 
  
 8080 
  
 targetPort 
 : 
  
 8080 
 --- 
 kind 
 : 
  
 ServiceExport 
 apiVersion 
 : 
  
 net.gke.io/v1 
  metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 store

The following diagram shows what happens after a ServiceExport is deployed. If a ServiceExport and Service pair exist then the multi-cluster Service controller deploys a corresponding ServiceImport to every GKE cluster in the fleet. The ServiceImport is the local representation of the store Service in every cluster. This enables the client Pod in gke-east-1 to use ClusterIP or headless Services to reach the store Pods in gke-west-1 . When used in this manner multi-cluster Services provide east-west load balancing between clusters without requiring an internal LoadBalancer Service. To use multi-cluster Services for cluster-to-cluster load balancing, see Configuring multi-cluster Services .

Multi-cluster Services exports Services across clusters which allows
cluster-to-cluster communication

Multi-cluster Gateways also use ServiceImports, but not for cluster-to-cluster load balancing. Instead, Gateways use ServiceImports as logical identifiers for a Service that exists in another cluster or that stretches across multiple clusters. The following HTTPRoute references a ServiceImport instead of a Service resource. By referencing a ServiceImport, this indicates that it is forwarding traffic to a group of backend Pods that run across one or more clusters.

  kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 store-route 
  
 namespace 
 : 
  
 store 
  
 labels 
 : 
  
 gateway 
 : 
  
 multi-cluster-gateway 
 spec 
 : 
  
 parentRefs 
 : 
  
 - 
  
 kind 
 : 
  
 Gateway 
  
 namespace 
 : 
  
 store 
  
 name 
 : 
  
 external-http 
  
 hostnames 
 : 
  
 - 
  
 "store.example.com" 
  
 rules 
 : 
  
  - 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store 
  
 port 
 : 
  
 8080

The following diagram shows how the HTTPRoute routes store.example.com traffic to store Pods on gke-west-1 and gke-east-1 . The load balancer treats them as one pool of backends. If the Pods from one of the clusters becomes unhealthy, unreachable, or has no traffic capacity, then traffic load is balanced to the remaining Pods on the other cluster. New clusters can be added or removed with the store Service and ServiceExport. This will transparently add or remove backend Pods without any explicit routing configuration changes.

MCS resource

Exporting Services

At this point, the application is running across both clusters. Next, you will expose and export the applications by deploying Services and ServiceExports to each cluster.

Apply the following manifest to the gke-west-1 cluster to create your store and store-west-1 Services and ServiceExports:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Service 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 selector 
 : 
  
 app 
 : 
  
 store 
  
 ports 
 : 
  
 - 
  
 port 
 : 
  
 8080 
  
 targetPort 
 : 
  
 8080 
 --- 
 kind 
 : 
  
 ServiceExport 
 apiVersion 
 : 
  
 net.gke.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 store 
 --- 
 apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Service 
 metadata 
 : 
  
 name 
 : 
  
 store-west-1 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 selector 
 : 
  
 app 
 : 
  
 store 
  
 ports 
 : 
  
 - 
  
 port 
 : 
  
 8080 
  
 targetPort 
 : 
  
 8080 
 --- 
 kind 
 : 
  
 ServiceExport 
 apiVersion 
 : 
  
 net.gke.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 store-west-1 
  
 namespace 
 : 
  
 store 
 EOF

Apply the following manifest to the gke-east-1 cluster to create your store and store-east-1 Services and ServiceExports:

  cat << EOF | kubectl apply --context gke-east-1 -f - 
 apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Service 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 selector 
 : 
  
 app 
 : 
  
 store 
  
 ports 
 : 
  
 - 
  
 port 
 : 
  
 8080 
  
 targetPort 
 : 
  
 8080 
 --- 
 kind 
 : 
  
 ServiceExport 
 apiVersion 
 : 
  
 net.gke.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 store 
 --- 
 apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Service 
 metadata 
 : 
  
 name 
 : 
  
 store-east-1 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 selector 
 : 
  
 app 
 : 
  
 store 
  
 ports 
 : 
  
 - 
  
 port 
 : 
  
 8080 
  
 targetPort 
 : 
  
 8080 
 --- 
 kind 
 : 
  
 ServiceExport 
 apiVersion 
 : 
  
 net.gke.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 store-east-1 
  
 namespace 
 : 
  
 store 
 EOF

Verify that the correct ServiceExports have been created in the clusters.
```
 kubectl  
get  
serviceexports  
--context  
 CLUSTER_NAME 
  
--namespace  
store 
```
Replace CLUSTER_NAME with gke-west-1 and gke-east-1 . The output should resemble the following:
```
  # gke-west-1 
NAME  
AGE
store  
2m40s
store-west-1  
2m40s # gke-east-1 
NAME  
AGE
store  
2m25s
store-east-1  
2m25s 
```
This demonstrates that the store Service contains store Pods across both clusters while the store-west-1 and store-east-1 Services only contain store Pods on their respective clusters. These overlapping Services are used to target the Pods across multiple clusters or a subset of Pods on a single cluster.
After a few minutes verify that the accompanying ServiceImports have been automatically created by the multi-cluster Services controller across all clusters in the fleet.

Note: The first MCS you create in your fleet can take up to 20 min to be fully operational. Exporting new services after the first one is created or adding endpoints to existing Multi-cluster Services is faster (up to a few minutes in some cases).
```
 kubectl  
get  
serviceimports  
--context  
 CLUSTER_NAME 
  
--namespace  
store 
```
Replace CLUSTER_NAME with gke-west-1 and gke-east-1 . The output should resemble the following:
```
  # gke-west-1 
NAME  
TYPE  
IP  
AGE
store  
ClusterSetIP  
 [ 
 "10.112.31.15" 
 ] 
  
6m54s
store-east-1  
ClusterSetIP  
 [ 
 "10.112.26.235" 
 ] 
  
5m49s
store-west-1  
ClusterSetIP  
 [ 
 "10.112.16.112" 
 ] 
  
6m54s # gke-east-1 
NAME  
TYPE  
IP  
AGE
store  
ClusterSetIP  
 [ 
 "10.72.28.226" 
 ] 
  
5d10h
store-east-1  
ClusterSetIP  
 [ 
 "10.72.19.177" 
 ] 
  
5d10h
store-west-1  
ClusterSetIP  
 [ 
 "10.72.28.68" 
 ] 
  
4h32m 
```
This demonstrates that all three Services are accessible from both clusters in the fleet. However, because there is only a single active config cluster per fleet, you can only deploy Gateways and HTTPRoutes that reference these ServiceImports in gke-west-1 . When an HTTPRoute in the config cluster references these ServiceImports as backends, the Gateway can forward traffic to these Services no matter which cluster they are exported from.

Deploying the Gateway and HTTPRoute

Once the applications have been deployed, you can then configure a Gateway using the gke-l7-global-external-managed-mc GatewayClass. This Gateway creates an external Application Load Balancer configured to distribute traffic across your target clusters.

Apply the following Gateway manifest to the config cluster, gke-west-1 in this example:
```
  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 Gateway 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 external-http 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 gatewayClassName 
 : 
  
 gke-l7-global-external-managed-mc 
  
 listeners 
 : 
  
 - 
  
 name 
 : 
  
 http 
  
 protocol 
 : 
  
 HTTP 
  
 port 
 : 
  
 80 
  
 allowedRoutes 
 : 
  
 kinds 
 : 
  
 - 
  
 kind 
 : 
  
 HTTPRoute 
 EOF 
 
```
Note: It might take several minutes (up to 10) for the Gateway to fully deploy and serve traffic.

This Gateway configuration deploys external Application Load Balancer resources with the following naming convention: gkemcg1- NAMESPACE - GATEWAY_NAME - HASH .

The default resources created with this configuration are:
- 1 load balancer: gkemcg1-store-external-http- HASH
- 1 public IP address: gkemcg1-store-external-http- HASH
- 1 forwarding rule: gkemcg1-store-external-http- HASH
- 2 backend services:
  - Default 404 backend service: gkemcg1-store-gw-serve404- HASH
  - Default 500 backend service: gkemcg1-store-gw-serve500- HASH
- 1 Health check:
  - Default 404 health check: gkemcg1-store-gw-serve404- HASH
- 0 routing rules (URLmap is empty)
At this stage, any request to the GATEWAY_IP :80 will result in a default page displaying the following message: fault filter abort .

Warning: In this example, we are deploying an external multi-cluster Gateway listening on port 80. If you are deploying an external multi-cluster Gateway for a production environment, you should ensure that you configure a set of features to properly secure your Gateway, depending on your context and your environment. To learn more about how to secure a multi-cluster Gateway, see Secure a Gateway , or how to apply Policies to a multi-cluster Gateway, see Configure Gateway resources using Policies .
Apply the following HTTPRoute manifest to the config cluster, gke-west-1 in this example:
```
  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 public-store-route 
  
 namespace 
 : 
  
 store 
  
 labels 
 : 
  
 gateway 
 : 
  
 external-http 
 spec 
 : 
  
 hostnames 
 : 
  
 - 
  
 "store.example.com" 
  
 parentRefs 
 : 
  
 - 
  
 name 
 : 
  
 external-http 
  
 rules 
 : 
  
 - 
  
 matches 
 : 
  
 - 
  
 path 
 : 
  
 type 
 : 
  
 PathPrefix 
  
 value 
 : 
  
 /west 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-west-1 
  
 port 
 : 
  
 8080 
  
 - 
  
 matches 
 : 
  
 - 
  
 path 
 : 
  
 type 
 : 
  
 PathPrefix 
  
 value 
 : 
  
 /east 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-east-1 
  
 port 
 : 
  
 8080 
  
 - 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store 
  
 port 
 : 
  
 8080 
 EOF 
 
```
At this stage, any request to the GATEWAY_IP :80 will result in a default page displaying the following message: fault filter abort .

Once deployed, this HTTPRoute will configure the following routing behavior:
- Requests to /west are routed to store Pods in the gke-west-1 cluster, because Pods selected by the store-west-1 ServiceExport only exist in the gke-west-1 cluster.
- Requests to /east are routed to store Pods in the gke-east-1 cluster, because Pods selected by the store-east-1 ServiceExport only exist in the gke-east-1 cluster.
- Requests to any other path are routed to store Pods in either cluster, according to its health, capacity, and proximity to the requesting client.
- Requests to the GATEWAY_IP :80 will result in a default page displaying the following message: fault filter abort .
Note that if all the Pods on a given cluster are unhealthy (or don't exist) then traffic to the store Service would only be sent to clusters that actually have store Pods. The existence of a ServiceExport and Service on a given cluster does not guarantee that traffic will be sent to that cluster. Pods must exist and be responding affirmatively to the load balancer health check or else the load balancer will just send traffic to healthy store Pods in other clusters.

New resources are created with this configuration:
- 3 backend services:
  - The store backend service: gkemcg1-store-store-8080- HASH
  - The store-east-1 backend service: gkemcg1-store-store-east-1-8080- HASH
  - The store-west-1 backend service: gkemcg1-store-store-west-1-8080- HASH
- 3 Health checks:
  - The store health check: gkemcg1-store-store-8080- HASH
  - The store-east-1 health check: gkemcg1-store-store-east-1-8080- HASH
  - The store-west-1 health check: gkemcg1-store-store-west-1-8080- HASH
- 1 routing rule in the URLmap:
  - The store.example.com routing rule:
  - 1 Host: store.example.com
  - Multiple matchRules to route to the new backend services

The following diagram shows the resources you've deployed across both clusters. Because gke-west-1 is the Gateway config cluster, it is the cluster in which our Gateway, HTTPRoutes, and ServiceImports are watched by the Gateway controller. Each cluster has a store ServiceImport and another ServiceImport specific to that cluster. Both point at the same Pods. This lets the HTTPRoute to specify exactly where traffic should go - to the store Pods on a specific cluster or to the store Pods across all clusters.

This is the Gateway and Multi-cluster Service resource model across both
clusters

Note that this is a logical resource model, not a depiction of the traffic flow. The traffic path goes directly from the load balancer to backend Pods and has no direct relation to whichever cluster is the config cluster.

Validating deployment

You can now issue requests to our multi-cluster Gateway and distribute traffic across both GKE clusters.

Validate that the Gateway and HTTPRoute have been deployed successfully by inspecting the Gateway status and events.

 kubectl  
describe  
gateways.gateway.networking.k8s.io  
external-http  
--context  
gke-west-1  
--namespace  
store

Your output should look similar to the following:

 Name:         external-http
Namespace:    store
Labels:       <none>
Annotations:  networking.gke.io/addresses: /projects/ PROJECT_NUMBER 
/global/addresses/gkemcg1-store-external-http-laup24msshu4
              networking.gke.io/backend-services:
                /projects/ PROJECT_NUMBER 
/global/backendServices/gkemcg1-store-gw-serve404-80-n65xmts4xvw2, /projects/ PROJECT_NUMBER 
/global/backendServices/gke...
              networking.gke.io/firewalls: /projects/ PROJECT_NUMBER 
/global/firewalls/gkemcg1-l7-default-global
              networking.gke.io/forwarding-rules: /projects/ PROJECT_NUMBER 
/global/forwardingRules/gkemcg1-store-external-http-a5et3e3itxsv
              networking.gke.io/health-checks:
                /projects/ PROJECT_NUMBER 
/global/healthChecks/gkemcg1-store-gw-serve404-80-n65xmts4xvw2, /projects/ PROJECT_NUMBER 
/global/healthChecks/gkemcg1-s...
              networking.gke.io/last-reconcile-time: 2023-10-12T17:54:24Z
              networking.gke.io/ssl-certificates:
              networking.gke.io/target-http-proxies: /projects/ PROJECT_NUMBER 
/global/targetHttpProxies/gkemcg1-store-external-http-94oqhkftu5yz
              networking.gke.io/target-https-proxies:
              networking.gke.io/url-maps: /projects/ PROJECT_NUMBER 
/global/urlMaps/gkemcg1-store-external-http-94oqhkftu5yz
API Version:  gateway.networking.k8s.io/v1beta1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2023-10-12T06:59:32Z
  Finalizers:
    gateway.finalizer.networking.gke.io
  Generation:        1
  Resource Version:  467057
  UID:               1dcb188e-2917-404f-9945-5f3c2e907b4c
Spec:
  Gateway Class Name:  gke-l7-global-external-managed-mc
  Listeners:
    Allowed Routes:
      Kinds:
        Group:  gateway.networking.k8s.io
        Kind:   HTTPRoute
      Namespaces:
        From:  Same
    Name:      http
    Port:      80
    Protocol:  HTTP
Status:
  Addresses:
    Type:   IPAddress
    Value:  34.36.127.249
  Conditions:
    Last Transition Time:  2023-10-12T07:00:41Z
    Message:               The OSS Gateway API has deprecated this condition, do not depend on it.
    Observed Generation:   1
    Reason:                Scheduled
    Status:                True
    Type:                  Scheduled
    Last Transition Time:  2023-10-12T07:00:41Z
    Message:
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Last Transition Time:  2023-10-12T07:00:41Z
    Message:
    Observed Generation:   1
    Reason:                Programmed
    Status:                True
    Type:                  Programmed
    Last Transition Time:  2023-10-12T07:00:41Z
    Message:               The OSS Gateway API has altered the "Ready" condition semantics and reservedit for future use.  GKE Gateway will stop emitting it in a future update, use "Programmed" instead.
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Listeners:
    Attached Routes:  1
    Conditions:
      Last Transition Time:  2023-10-12T07:00:41Z
      Message:
      Observed Generation:   1
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
      Last Transition Time:  2023-10-12T07:00:41Z
      Message:               The OSS Gateway API has altered the "Ready" condition semantics and reservedit for future use.  GKE Gateway will stop emitting it in a future update, use "Programmed" instead.
      Observed Generation:   1
      Reason:                Ready
      Status:                True
      Type:                  Ready
    Name:                    http
    Supported Kinds:
      Group:  gateway.networking.k8s.io
      Kind:   HTTPRoute
Events:
  Type    Reason  Age                    From                   Message
  ----    ------  ----                   ----                   -------
  Normal  UPDATE  35m (x4 over 10h)      mc-gateway-controller  store/external-http
  Normal  SYNC    4m22s (x216 over 10h)  mc-gateway-controller  SYNC on store/external-http was a success

Once the Gateway has deployed successfully retrieve the external IP address from external-http Gateway.

 kubectl  
get  
gateways.gateway.networking.k8s.io  
external-http  
-o = 
 jsonpath 
 = 
 "{.status.addresses[0].value}" 
  
--context  
gke-west-1  
--namespace  
store

Replace VIP in the following steps with the IP address you receive as output.

Send traffic to the root path of the domain. This load balances traffic to the store ServiceImport which is across cluster gke-west-1 and gke-east-1 . The load balancer sends your traffic to the closest region to you and you might not see responses from the other region.

 curl  
-H  
 "host: store.example.com" 
  
http:// VIP

The output confirms that the request was served by Pod from the gke-east-1 cluster:

  { 
  
  "cluster_name" 
 : 
  
 "gke-east-1" 
 , 
  
  "zone" 
 : 
  
 "us-east1-b" 
 , 
  
 "host_header" 
 : 
  
 "store.example.com" 
 , 
  
 "node_name" 
 : 
  
 "gke-gke-east-1-default-pool-7aa30992-t2lp.c.agmsb-k8s.internal" 
 , 
  
 "pod_name" 
 : 
  
 "store-5f5b954888-dg22z" 
 , 
  
 "pod_name_emoji" 
 : 
  
 "⏭" 
 , 
  
 "project_id" 
 : 
  
 "agmsb-k8s" 
 , 
  
 "timestamp" 
 : 
  
 "2021-06-01T17:32:51" 
 }

Next send traffic to the /west path. This routes traffic to the store-west-1 ServiceImport which only has Pods running on the gke-west-1 cluster. A cluster-specific ServiceImport, like store-west-1 , enables an application owner to explicitly send traffic to a specific cluster, rather than letting the load balancer make the decision.

 curl  
-H  
 "host: store.example.com" 
  
http:// VIP 
/west

The output confirms that the request was served by Pod from the gke-west-1 cluster:

  { 
  
  "cluster_name" 
 : 
  
 "gke-west-1" 
 , 
  
  
  "zone" 
 : 
  
 "us-west1-a" 
 , 
  
  
 "host_header" 
 : 
  
 "store.example.com" 
 , 
  
 "node_name" 
 : 
  
 "gke-gke-west-1-default-pool-65059399-2f41.c.agmsb-k8s.internal" 
 , 
  
 "pod_name" 
 : 
  
 "store-5f5b954888-d25m5" 
 , 
  
 "pod_name_emoji" 
 : 
  
 "🍾" 
 , 
  
 "project_id" 
 : 
  
 "agmsb-k8s" 
 , 
  
 "timestamp" 
 : 
  
 "2021-06-01T17:39:15" 
 , 
 }

Finally, send traffic to the /east path.

 curl  
-H  
 "host: store.example.com" 
  
http:// VIP 
/east

The output confirms that the request was served by Pod from the gke-east-1 cluster:

  { 
  
  "cluster_name" 
 : 
  
 "gke-east-1" 
 , 
  
  "zone" 
 : 
  
 "us-east1-b" 
 , 
  
 "host_header" 
 : 
  
 "store.example.com" 
 , 
  
 "node_name" 
 : 
  
 "gke-gke-east-1-default-pool-7aa30992-7j7z.c.agmsb-k8s.internal" 
 , 
  
 "pod_name" 
 : 
  
 "store-5f5b954888-hz6mw" 
 , 
  
 "pod_name_emoji" 
 : 
  
 "🧜🏾" 
 , 
  
 "project_id" 
 : 
  
 "agmsb-k8s" 
 , 
  
 "timestamp" 
 : 
  
 "2021-06-01T17:40:48" 
 }

Deploy an internal multi-cluster Gateway across regions

You can deploy multi-cluster Gateways that provide internal Layer 7 load balancing across GKE clusters in multiple regions. These Gateways use the gke-l7-cross-regional-internal-managed-mc GatewayClass. This GatewayClass provisions a cross-region internal Application Load Balancer that's managed by Google Cloud and that enables internal VIPs that clients within your VPC network can access. These Gateways can be exposed by frontends in the regions of your choice, simply by using the Gateway to request addresses in those regions. The internal VIP can be a single IP address, or they can be IP addresses in multiple regions, with one IP address per region that's specified in the Gateway. Traffic is directed to the closest healthy backend GKE cluster that can serve the request.

Before you begin

Set up your project and shell by configuring your gcloud environment with your project ID:

  export 
  
 PROJECT_ID 
 = 
 " YOUR_PROJECT_ID 
" 
gcloud  
config  
 set 
  
project  
 ${ 
 PROJECT_ID 
 }

Create GKE clusters in different regions.

This example uses two clusters, gke-west-1 in us-west1 and gke-east-1 in us-east1 . Ensure the Gateway API is enabled ( --gateway-api=standard ) and clusters are registered to a fleet.

 gcloud  
container  
clusters  
create  
gke-west-1  
 \ 
  
--location = 
us-west1-a  
 \ 
  
--workload-pool = 
 ${ 
 PROJECT_ID 
 } 
.svc.id.goog  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
  
 \ 
  
--enable-fleet  
 \ 
  
--gateway-api = 
standard

gcloud  
container  
clusters  
create  
gke-east-1  
 \ 
  
--location = 
us-east1-c  
 \ 
  
--workload-pool = 
 ${ 
 PROJECT_ID 
 } 
.svc.id.goog  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
  
 \ 
  
--enable-fleet  
 \ 
  
--gateway-api = 
standard

Rename contexts for easier access:

 gcloud  
container  
clusters  
get-credentials  
gke-west-1  
 \ 
  
--location = 
us-west1-a  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
gcloud  
container  
clusters  
get-credentials  
gke-east-1  
 \ 
  
--location = 
us-east1-c  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
kubectl  
config  
rename-context  
gke_ ${ 
 PROJECT_ID 
 } 
_us-west1-a_gke-west-1  
gke-west1
kubectl  
config  
rename-context  
gke_ ${ 
 PROJECT_ID 
 } 
_us-east1-c_gke-east-1  
gke-east1

Enable Multi-Cluster Services (MCS) and Multi-Cluster Ingress (MCI/Gateway):

 gcloud  
container  
fleet  
multi-cluster-services  
 enable 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
 # Set the config membership to one of your clusters (e.g., gke-west-1) 
 # This cluster will be the source of truth for multi-cluster Gateway and Route resources. 
gcloud  
container  
fleet  
ingress  
 enable 
  
 \ 
  
--config-membership = 
projects/ ${ 
 PROJECT_ID 
 } 
/locations/us-west1/memberships/gke-west-1  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 }

Configure proxy-only subnets. A proxy-only subnet is required in each region where your GKE clusters are located and where the load balancer will operate. Cross-Region internal internal Application Load Balancer require the purpose of this subnet to be set to GLOBAL_MANAGED_PROXY .

  # Proxy-only subnet for us-west1 
gcloud  
compute  
networks  
subnets  
create  
us-west1-proxy-only-subnet  
 \ 
  
--purpose = 
GLOBAL_MANAGED_PROXY  
 \ 
  
--role = 
ACTIVE  
 \ 
  
--region = 
us-west1  
 \ 
  
--network = 
default  
 \ 
  
--range = 
 10 
.129.0.0/23  
 # Choose an appropriate unused CIDR range 
 # Proxy-only subnet for us-east1 
gcloud  
compute  
networks  
subnets  
create  
us-east1-proxy-only-subnet  
 \ 
  
--purpose = 
GLOBAL_MANAGED_PROXY  
 \ 
  
--role = 
ACTIVE  
 \ 
  
--region = 
us-east1  
 \ 
  
--network = 
default  
 \ 
  
--range = 
 10 
.130.0.0/23  
 # Choose an appropriate unused CIDR range

If you're not using the default network, replace default with the name of your VPC network. Ensure that the CIDR ranges are unique and don't overlap.

Deploy your demo applications, such as store , to both clusters. The example store.yaml file from gke-networking-recipes creates a store namespace and a deployment.

 kubectl  
apply  
--context  
gke-west1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml
kubectl  
apply  
--context  
gke-east1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml

Export Services from each cluster by creating Kubernetes Service resources and ServiceExport resources in each cluster, which makes the services discoverable across the fleet. The following example exports a generic store service and region-specific services ( store-west-1 , store-east-1 ) from each cluster, all within the store namespace.

Apply to gke-west1 :

 cat << 
EOF  
 | 
  
kubectl  
apply  
--context  
gke-west1  
-f  
-
apiVersion:  
v1
kind:  
Service
metadata:  
name:  
store  
namespace:  
store
spec:  
selector:  
app:  
store  
ports:  
-  
port:  
 8080 
  
targetPort:  
 8080 
---
kind:  
ServiceExport
apiVersion:  
net.gke.io/v1
metadata:  
name:  
store  
namespace:  
store
---
apiVersion:  
v1
kind:  
Service
metadata:  
name:  
store-west-1  
 # Specific to this cluster 
  
namespace:  
store
spec:  
selector:  
app:  
store  
ports:  
-  
port:  
 8080 
  
targetPort:  
 8080 
---
kind:  
ServiceExport
apiVersion:  
net.gke.io/v1
metadata:  
name:  
store-west-1  
 # Exporting the region-specific service 
  
namespace:  
store
EOF

Apply to gke-east1 :

 cat << 
EOF  
 | 
  
kubectl  
apply  
--context  
gke-east1  
-f  
-
apiVersion:  
v1
kind:  
Service
metadata:  
name:  
store  
namespace:  
store
spec:  
selector:  
app:  
store  
ports:  
-  
port:  
 8080 
  
targetPort:  
 8080 
---
kind:  
ServiceExport
apiVersion:  
net.gke.io/v1
metadata:  
name:  
store  
namespace:  
store
---
apiVersion:  
v1
kind:  
Service
metadata:  
name:  
store-east-1  
 # Specific to this cluster 
  
namespace:  
store
spec:  
selector:  
app:  
store  
ports:  
-  
port:  
 8080 
  
targetPort:  
 8080 
---
kind:  
ServiceExport
apiVersion:  
net.gke.io/v1
metadata:  
name:  
store-east-1  
 # Exporting the region-specific service 
  
namespace:  
store
EOF

Check ServiceImports: Verify that ServiceImport resources are created in each cluster within the store namespace. It might take a few minutes for them to be created. bash kubectl get serviceimports --context gke-west1 -n store kubectl get serviceimports --context gke-east1 -n store You should see store , store-west-1 , and store-east-1 listed (or relevant entries based on propagation).

Configure an internal multi-region Gateway

Define a Gateway resource that references the gke-l7-cross-regional-internal-managed-mc GatewayClass. You apply this manifest to your designated config cluster, such as gke-west-1 .

The spec.addresses field lets you request ephemeral IP addresses in specific regions or use pre-allocated static IP addresses.

To use ephemeral IP addresses, save the following Gateway manifest as cross-regional-gateway.yaml :

  # cross-regional-gateway.yaml 
 kind 
 : 
  
 Gateway 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 internal-cross-region-gateway 
  
 namespace 
 : 
  
 store 
  
 # Namespace for the Gateway resource 
 spec 
 : 
  
 gatewayClassName 
 : 
  
 gke-l7-cross-regional-internal-managed-mc 
  
 addresses 
 : 
  
 # Addresses across regions. Address value is allowed to be empty or matching 
  
 # the region name. 
  
 - 
  
 type 
 : 
  
 networking.gke.io/ephemeral-ipv4-address/us-west1 
  
 value 
 : 
  
 "us-west1" 
  
 - 
  
 type 
 : 
  
 networking.gke.io/ephemeral-ipv4-address/us-east1 
  
 value 
 : 
  
 "us-east1" 
  
 listeners 
 : 
  
 - 
  
 name 
 : 
  
 http 
  
 protocol 
 : 
  
 HTTP 
  
 port 
 : 
  
 80 
  
 allowedRoutes 
 : 
  
 kinds 
 : 
  
 - 
  
 kind 
 : 
  
 HTTPRoute 
  
 # Only allow HTTPRoute to attach

The following list defines some of the fields in the previous YAML file:

metadata.namespace : the namespace where the Gateway resource is created, for example, store .
spec.gatewayClassName : the name of the GatewayClass. Must be gke-l7-cross-regional-internal-managed-mc .
spec.listeners.allowedRoutes.kinds : the kinds of Route objects that can be attached, for example, HTTPRoute .
spec.addresses :
- type: networking.gke.io/ephemeral-ipv4-address/ REGION : requests an ephemeral IP address.
- value : specifies the region for the address, for example, "us-west1" or "us-east1" .

Apply the manifest to your config cluster, for example, gke-west1 :

 kubectl  
apply  
--context  
gke-west1  
-f  
cross-regional-gateway.yaml

Attach HTTPRoutes to the Gateway

Define HTTPRoute resources to manage traffic routing and apply them to your config cluster.

Save the following HTTPRoute manifest as store-route.yaml :

  # store-route.yaml 
 kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 store-route 
  
 namespace 
 : 
  
 store 
  
 labels 
 : 
  
 gateway 
 : 
  
 cross-regional-internal 
 spec 
 : 
  
 parentRefs 
 : 
  
 - 
  
 name 
 : 
  
 internal-cross-region-gateway 
  
 namespace 
 : 
  
 store 
  
 # Namespace where the Gateway is deployed 
  
 hostnames 
 : 
  
 - 
  
 "store.example.internal" 
  
 # Hostname clients will use 
  
 rules 
 : 
  
 - 
  
 matches 
 : 
  
 # Rule for traffic to /west 
  
 - 
  
 path 
 : 
  
 type 
 : 
  
 PathPrefix 
  
 value 
 : 
  
 /west 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 # Indicates a multi-cluster ServiceImport 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-west-1 
  
 # Targets the ServiceImport for the west cluster 
  
 port 
 : 
  
 8080 
  
 - 
  
 matches 
 : 
  
 # Rule for traffic to /east 
  
 - 
  
 path 
 : 
  
 type 
 : 
  
 PathPrefix 
  
 value 
 : 
  
 /east 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-east-1 
  
 # Targets the ServiceImport for the east cluster 
  
 port 
 : 
  
 8080 
  
 - 
  
 backendRefs 
 : 
  
 # Default rule for other paths (e.g., /) 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store 
  
 # Targets the generic 'store' ServiceImport (any region) 
  
 port 
 : 
  
 8080

The following list defines some of the fields in the previous YAML file:

spec.parentRefs : attaches this route to internal-cross-region-gateway in the store namespace.
spec.hostnames : represents the hostname that clients use to access the service.
spec.rules : defines routing logic. This example uses path-based routing:
- /west traffic goes to store-west-1 ServiceImport.
- /east traffic goes to store-east-1 ServiceImport.
- All other traffic, such as / , goes to the generic store ServiceImport.
backendRefs :
- group: net.gke.io and kind: ServiceImport target multi-cluster services.

Apply the HTTPRoute manifest to your config cluster:

 kubectl  
apply  
--context  
gke-west1  
-f  
store-route.yaml

Verify the status of the Gateway and Route

Check the Gateway status:
```
 kubectl  
get  
gateway  
internal-cross-region-gateway  
-n  
store  
-o  
yaml  
--context  
gke-west1 
```
Look for a condition with type: Programmed and status: "True" . You should see IP addresses assigned in the status.addresses field, corresponding to the regions you specified (e.g., one for us-west1 and one for us-east1`).
Check the HTTPRoute status:
```
 kubectl  
get  
httproute  
store-route  
-n  
store  
-o  
yaml  
--context  
gke-west1 
```
Look for a condition in status.parents[].conditions with type: Accepted (or ResolvedRefs ) and status: "True" .

Confirm traffic

After you assign the IP addresses to the Gateway, you can test traffic from a client VM that's within your VPC network and in one of the regions, or in a region that can connect to the Gateway IP address.

Retrieve the Gateway IP addresses.

The following command attempts to parse the JSON output. You might need to adjust the jsonpath based on the exact structure.
```
 kubectl  
get  
gateway  
cross-region-gateway  
-n  
store  
--context  
gke-west1  
-o = 
 jsonpath 
 = 
 "{.status.addresses[*].value}" 
. 
```
The output of this command should include the VIPs, such as VIP1_WEST , or VIP2_EAST .

Send test requests: From a client VM in your VPC:

  # Assuming VIP_WEST is an IP in us-west1 and VIP_EAST is an IP in us-east1 
 # Traffic to /west should ideally be served by gke-west-1 
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_WEST/west
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_EAST/west  
 # Still targets store-west-1 due to path 
 # Traffic to /east should ideally be served by gke-east-1 
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_WEST/east  
 # Still targets store-east-1 due to path 
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_EAST/east # Traffic to / (default) could be served by either cluster 
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_WEST/
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_EAST/

The response should include details from the store application that indicate which backend pod served the request, such as cluster_name or zone .

Use static IP Addresses

Instead of ephemeral IP addresses, you can use pre-allocated static internal IP addresses.

Create static IP addresses in the regions that you want to use:

 gcloud  
compute  
addresses  
create  
cross-region-gw-ip-west  
--region  
us-west1  
--subnet  
default  
--project = 
 ${ 
 PROJECT_ID 
 } 
gcloud  
compute  
addresses  
create  
cross-region-gw-ip-east  
--region  
us-east1  
--subnet  
default  
--project = 
 ${ 
 PROJECT_ID 
 }

If you're not using the default subnet, replace default with the name of the subnet that has the IP address you want to allocate. These subnets are regular subnets, not the proxy-only subnets.

Update the Gateway manifest by modifying the spec.addresses section in your cross-regional-gateway.yaml file:

  # cross-regional-gateway-static-ip.yaml 
 kind 
 : 
  
 Gateway 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 internal-cross-region-gateway 
  
 # Or a new name if deploying alongside 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 gatewayClassName 
 : 
  
 gke-l7-cross-regional-internal-managed-mc 
  
 addresses 
 : 
  
 - 
  
 type 
 : 
  
 networking.gke.io/named-address-with-region 
  
 # Use for named static IP 
  
 value 
 : 
  
 "regions/us-west1/addresses/cross-region-gw-ip-west" 
  
 - 
  
 type 
 : 
  
 networking.gke.io/named-address-with-region 
  
 value 
 : 
  
 "regions/us-east1/addresses/cross-region-gw-ip-east" 
  
 listeners 
 : 
  
 - 
  
 name 
 : 
  
 http 
  
 protocol 
 : 
  
 HTTP 
  
 port 
 : 
  
 80 
  
 allowedRoutes 
 : 
  
 kinds 
 : 
  
 - 
  
 kind 
 : 
  
 HTTPRoute

Apply the updated Gateway manifest.

 kubectl  
apply  
--context  
gke-west1  
-f  
cross-regional-gateway.yaml

Special considerations for non-default subnets

Be aware of the following considerations when you use non-default subnets:

Same VPC network:all user-created resources—such as static IP addresses, proxy-only subnets, and GKE clusters—must reside within the same VPC network.
Address subnet:when you create static IP addresses for the Gateway, they are allocated from regular subnets in the specified regions.
Cluster subnet naming:Each region must have a subnet that has the same name as the subnet that the MCG config cluster resides in.
- For example, if your gke-west-1 config cluster is in projects/YOUR_PROJECT/regions/us-west1/subnetworks/my-custom-subnet , then the regions you are requesting addresses for must also have the my-custom-subnet subnet. If you request addresses in the us-east1 and us-centra1 regions, then a subnet named my-custom-subnet must also exist in those regions.

Blue-green, multi-cluster routing with Gateway

The gke-l7-global-external-managed-* , gke-l7-regional-external-managed-* , and gke-l7-rilb-* GatewayClasses have many advanced traffic routing capabilities including traffic splitting, header matching, header manipulation, traffic mirroring, and more. In this example, you'll demonstrate how to use weight-based traffic splitting to explicitly control the traffic proportion across two GKE clusters.

This example goes through some realistic steps that a service owner would take in moving or expanding their application to a new GKE cluster. The goal of blue-green deployments is to reduce risk through multiple validation steps which confirm that the new cluster is operating correctly. This example walks through four stages of deployment:

100% - Header-based canary : Use HTTP header routing to send only test or synthetic traffic to the new cluster.
100% - Mirror traffic : Mirror user traffic to the canary cluster. This tests the capacity of the canary cluster by copying 100% of the user traffic to this cluster.
90% - 10% : Canary a traffic split of 10% to slowly expose the new cluster to live traffic.
0% - 100% : Cutover fully to the new cluster with the option of switching back if any errors are observed.

Blue-green traffic splitting across two GKE clusters

This example is similar to the previous one, except it deploys an internal multi-cluster Gateway instead. This deploys an internal Application Load Balancer which is only privately accessible from within the VPC. You will use the clusters and same application that you deployed in the previous steps, except deploy them through a different Gateway.

Prerequisites

The following example builds on some of the steps in Deploying an external multi-cluster Gateway . Ensure that you have done the following steps before proceeding with this example:

Enabling multi-cluster Gateways
Deploying a demo application

This example uses the gke-west-1 and gke-west-2 clusters that you already set up. These clusters are in the same region because the gke-l7-rilb-mc GatewayClass is regional and only supports cluster backends in the same region.

Deploy the Service and ServiceExports needed on each cluster. If you deployed Services and ServiceExports from the previous example then you already deployed some of these.

 kubectl  
apply  
--context  
gke-west-1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store-west-1-service.yaml
kubectl  
apply  
--context  
gke-west-2  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store-west-2-service.yaml

It deploys a similar set of resources to each cluster:

 service/store created
serviceexport.net.gke.io/store created
service/store-west-2 created
serviceexport.net.gke.io/store-west-2 created

Configuring a proxy-only subnet

If you have not already done so, configure a proxy-only subnet for each region in which you are deploying internal Gateways. This subnet is used to provide internal IP addresses to the load balancer proxies and must be configured with a --purpose set to REGIONAL_MANAGED_PROXY only.

You must create a proxy-only subnet before you create Gateways that manage internal Application Load Balancers. Each region of a Virtual Private Cloud (VPC) network in which you use internal Application Load Balancers must have a proxy-only subnet.

The gcloud compute networks subnets create command creates a proxy-only a subnet.

 gcloud  
compute  
networks  
subnets  
create  
 SUBNET_NAME 
  
 \ 
  
--purpose = 
REGIONAL_MANAGED_PROXY  
 \ 
  
--role = 
ACTIVE  
 \ 
  
--region = 
 REGION 
  
 \ 
  
--network = 
 VPC_NETWORK_NAME 
  
 \ 
  
--range = 
 CIDR_RANGE

Replace the following:

SUBNET_NAME : the name of the proxy-only subnet.
REGION : the region of the proxy-only subnet.
VPC_NETWORK_NAME : the name of the VPC network that contains the subnet.
CIDR_RANGE : the primary IP address range of the subnet. You must use a subnet mask no larger than /26 so that at least 64 IP addresses are available for proxies in the region. The recommended subnet mask is /23 .

Deploying the Gateway

The following Gateway is created from the gke-l7-rilb-mc GatewayClass, which is a regional internal Gateway that can target only GKE clusters in the same region.

Apply the following Gateway manifest to the config cluster, gke-west-1 in this example:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 Gateway 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 internal-http 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 gatewayClassName 
 : 
  
 gke-l7-rilb-mc 
  
 listeners 
 : 
  
 - 
  
 name 
 : 
  
 http 
  
 protocol 
 : 
  
 HTTP 
  
 port 
 : 
  
 80 
  
 allowedRoutes 
 : 
  
 kinds 
 : 
  
 - 
  
 kind 
 : 
  
 HTTPRoute 
 EOF

Validate that the Gateway has come up successfully. You can filter for just the events from this Gateway with the following command:

 kubectl  
get  
events  
--field-selector  
involvedObject.kind = 
Gateway,involvedObject.name = 
internal-http  
--context = 
gke-west-1  
--namespace  
store

The Gateway deployment was successful if the output resembles the following:

 LAST SEEN   TYPE     REASON   OBJECT                  MESSAGE
5m18s       Normal   ADD      gateway/internal-http   store/internal-http
3m44s       Normal   UPDATE   gateway/internal-http   store/internal-http
3m9s        Normal   SYNC     gateway/internal-http   SYNC on store/internal-http was a success

Header-based canary

Header-based canarying lets the service owner match synthetic test traffic that does not come from real users. This is an easy way of validating that the basic networking of the application is functioning without exposing users directly.

Apply the following HTTPRoute manifest to the config cluster, gke-west-1 in this example:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 internal-store-route 
  
 namespace 
 : 
  
 store 
  
 labels 
 : 
  
 gateway 
 : 
  
 internal-http 
 spec 
 : 
  
 parentRefs 
 : 
  
 - 
  
 kind 
 : 
  
 Gateway 
  
 namespace 
 : 
  
 store 
  
 name 
 : 
  
 internal-http 
  
 hostnames 
 : 
  
 - 
  
 "store.example.internal" 
  
 rules 
 : 
  
 # Matches for env=canary and sends it to store-west-2 ServiceImport 
  
 - 
  
 matches 
 : 
  
 - 
  
 headers 
 : 
  
 - 
  
 name 
 : 
  
 env 
  
 value 
 : 
  
 canary 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-west-2 
  
 port 
 : 
  
 8080 
  
 # All other traffic goes to store-west-1 ServiceImport 
  
 - 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-west-1 
  
 port 
 : 
  
 8080 
 EOF

Once deployed, this HTTPRoute configures the following routing behavior:

Internal requests to store.example.internal withoutthe env: canary HTTP header are routed to store Pods on the gke-west-1 cluster
Internal requests to store.example.internal withthe env: canary HTTP header are routed to store Pods on the gke-west-2 cluster

The HTTPRoute enables routing to different clusters based on the HTTP
headers

Validate that the HTTPRoute is functioning correctly by sending traffic to the Gateway IP address.

Retrieve the internal IP address from internal-http .
```
 kubectl  
get  
gateways.gateway.networking.k8s.io  
internal-http  
-o = 
 jsonpath 
 = 
 "{.status.addresses[0].value}" 
  
--context  
gke-west-1  
--namespace  
store 
```
Replace VIP in the following steps with the IP address you receive as output.

Note: In the following steps send all requests to VIP from a client that has internal VPC connectivity and is in the same region as the GKE cluster (unless you have configured global access on your Gateway). You can create a VM in us-west1 and SSH to it for this purpose. This is necessary because the internal-http Gateway is an internal, regional load balancer. Also set the host header with store.example.internal so that DNS does not have to be configured for this example to work.

Send a request to the Gateway using the env: canary HTTP header. This will confirm that traffic is being routed to gke-west-2 . Use a private client in the same VPC as the GKE clusters to confirm that requests are being routed correctly. The following command must be run on a machine that has private access to the Gateway IP address or else it will not function.

 curl  
-H  
 "host: store.example.internal" 
  
-H  
 "env: canary" 
  
http:// VIP

The output confirms that the request was served by a Pod from the gke-west-2 cluster:

  { 
  
  "cluster_name" 
 : 
  
 "gke-west-2" 
 , 
  
  
 "host_header" 
 : 
  
 "store.example.internal" 
 , 
  
 "node_name" 
 : 
  
 "gke-gke-west-2-default-pool-4cde1f72-m82p.c.agmsb-k8s.internal" 
 , 
  
 "pod_name" 
 : 
  
 "store-5f5b954888-9kdb5" 
 , 
  
 "pod_name_emoji" 
 : 
  
 "😂" 
 , 
  
 "project_id" 
 : 
  
 "agmsb-k8s" 
 , 
  
 "timestamp" 
 : 
  
 "2021-05-31T01:21:55" 
 , 
  
 "zone" 
 : 
  
 "us-west1-a" 
 }

Traffic mirror

This stage sends traffic to the intended cluster but also mirrors that traffic to the canary cluster.

Using mirroring is helpful to determine how traffic load will impact application performance without impacting responses to your clients in any way. It may not be necessary for all kinds of rollouts, but can be useful when rolling out large changes that could impact performance or load.

Apply the following HTTPRoute manifest to the config cluster, gke-west-1 in this example:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 internal-store-route 
  
 namespace 
 : 
  
 store 
  
 labels 
 : 
  
 gateway 
 : 
  
 internal-http 
 spec 
 : 
  
 parentRefs 
 : 
  
 - 
  
 kind 
 : 
  
 Gateway 
  
 namespace 
 : 
  
 store 
  
 name 
 : 
  
 internal-http 
  
 hostnames 
 : 
  
 - 
  
 "store.example.internal" 
  
 rules 
 : 
  
 # Sends all traffic to store-west-1 ServiceImport 
  
 - 
  
 backendRefs 
 : 
  
 - 
  
 name 
 : 
  
 store-west-1 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 port 
 : 
  
 8080 
  
 # Also mirrors all traffic to store-west-2 ServiceImport 
  
 filters 
 : 
  
 - 
  
 type 
 : 
  
 RequestMirror 
  
 requestMirror 
 : 
  
 backendRef 
 : 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-west-2 
  
 port 
 : 
  
 8080 
 EOF

Using your private client, send a request to the internal-http Gateway. Use the /mirror path so you can uniquely identify this request in the application logs in a later step.
```
 curl  
-H  
 "host: store.example.internal" 
  
http:// VIP 
/mirror 
```

The output confirms that the client received a response from a Pod in the gke-west-1 cluster:

  { 
  
  "cluster_name" 
 : 
  
 "gke-west-1" 
 , 
  
  
 "host_header" 
 : 
  
 "store.example.internal" 
 , 
  
 "node_name" 
 : 
  
 "gke-gke-west-1-default-pool-65059399-ssfq.c.agmsb-k8s.internal" 
 , 
  
 "pod_name" 
 : 
  
 "store-5f5b954888-brg5w" 
 , 
  
 "pod_name_emoji" 
 : 
  
 "🎖" 
 , 
  
 "project_id" 
 : 
  
 "agmsb-k8s" 
 , 
  
 "timestamp" 
 : 
  
 "2021-05-31T01:24:51" 
 , 
  
 "zone" 
 : 
  
 "us-west1-a" 
 }

This confirms that the primary cluster is responding to traffic. You still need to confirm that the cluster you are migrating to is receiving mirrored traffic.

Check the application logs of a store Pod on the gke-west-2 cluster. The logs should confirm that the Pod received mirrored traffic from the load balancer.
```
 kubectl  
logs  
deployment/store  
--context  
gke-west-2  
-n  
store  
 | 
  
grep  
/mirror 
```

This output confirms that Pods on the gke-west-2 cluster are also receiving the same requests, however their responses to these requests are not sent back to the client. The IP addresses seen in the logs are that of the load balancer's internal IP addresses which are communicating with your Pods.

 Found  
 2 
  
pods,  
using  
pod/store-5c65bdf74f-vpqbs [ 
 2023 
-10-12  
 21 
:05:20,805 ] 
  
INFO  
 in 
  
_internal:  
 192 
.168.21.3  
-  
-  
 [ 
 12 
/Oct/2023  
 21 
:05:20 ] 
  
  "GET /mirror HTTP/1.1" 
  
 200 
  
- [ 
 2023 
-10-12  
 21 
:05:27,158 ] 
  
INFO  
 in 
  
_internal:  
 192 
.168.21.3  
-  
-  
 [ 
 12 
/Oct/2023  
 21 
:05:27 ] 
  
  "GET /mirror HTTP/1.1" 
  
 200 
  
- [ 
 2023 
-10-12  
 21 
:05:27,805 ] 
  
INFO  
 in 
  
_internal:  
 192 
.168.21.3  
-  
-  
 [ 
 12 
/Oct/2023  
 21 
:05:27 ] 
  
  "GET /mirror HTTP/1.1" 
  
 200 
  
-

Traffic split

Traffic splitting is one of the most common methods of rolling out new code or deploying to new environments safely. The service owner sets an explicit percentage of traffic that is sent to the canary backends that is typically a very small amount of the overall traffic so that the success of the rollout can be determined with an acceptable amount of risk to real user requests.

Doing a traffic split with a minority of the traffic enables the service owner to inspect the health of the application and the responses. If all the signals look healthy, then they may proceed to the full cutover.

Apply the following HTTPRoute manifest to the config cluster, gke-west-1 in this example:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 internal-store-route 
  
 namespace 
 : 
  
 store 
  
 labels 
 : 
  
 gateway 
 : 
  
 internal-http 
 spec 
 : 
  
 parentRefs 
 : 
  
 - 
  
 kind 
 : 
  
 Gateway 
  
 namespace 
 : 
  
 store 
  
 name 
 : 
  
 internal-http 
  
 hostnames 
 : 
  
 - 
  
 "store.example.internal" 
  
 rules 
 : 
  
 - 
  
 backendRefs 
 : 
  
 # 90% of traffic to store-west-1 ServiceImport 
  
 - 
  
 name 
 : 
  
 store-west-1 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 port 
 : 
  
 8080 
  
 weight 
 : 
  
 90 
  
 # 10% of traffic to store-west-2 ServiceImport 
  
 - 
  
 name 
 : 
  
 store-west-2 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 port 
 : 
  
 8080 
  
 weight 
 : 
  
 10 
 EOF

Using your private client, send a continuous curl request to the internal- http Gateway.

  while 
  
true ; 
  
 do 
  
curl  
-H  
 "host: store.example.internal" 
  
-s  
 VIP 
  
 | 
  
grep  
 "cluster_name" 
 ; 
  
sleep  
 1 
 ; 
  
 done

The output will be similar to this, indicating that a 90/10 traffic split is occurring.

 "cluster_name": "gke-west-1",
"cluster_name": "gke-west-1",
"cluster_name": "gke-west-1",
"cluster_name": "gke-west-1",
"cluster_name": "gke-west-1",
"cluster_name": "gke-west-1",
"cluster_name": "gke-west-1",
"cluster_name": "gke-west-1", "cluster_name": "gke-west-2","cluster_name": "gke-west-1",
"cluster_name": "gke-west-1",
...

Traffic cut over

The last stage of the blue-green migration is to fully cut over to the new cluster and remove the old cluster. If the service owner was actually onboarding a second cluster to an existing cluster then this last step would be different as the final step would have traffic going to both clusters. In that scenario a single store ServiceImport is recommended that has Pods from both gke-west-1 and gke-west-2 clusters. This allows the load balancer to make the decision of where traffic should go for an active-active application, based on proximity, health, and capacity.

Apply the following HTTPRoute manifest to the config cluster, gke-west-1 in this example:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 internal-store-route 
  
 namespace 
 : 
  
 store 
  
 labels 
 : 
  
 gateway 
 : 
  
 internal-http 
 spec 
 : 
  
 parentRefs 
 : 
  
 - 
  
 kind 
 : 
  
 Gateway 
  
 namespace 
 : 
  
 store 
  
 name 
 : 
  
 internal-http 
  
 hostnames 
 : 
  
 - 
  
 "store.example.internal" 
  
 rules 
 : 
  
 - 
  
 backendRefs 
 : 
  
 # No traffic to the store-west-1 ServiceImport 
  
 - 
  
 name 
 : 
  
 store-west-1 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 port 
 : 
  
 8080 
  
 weight 
 : 
  
 0 
  
 # All traffic to the store-west-2 ServiceImport 
  
 - 
  
 name 
 : 
  
 store-west-2 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 port 
 : 
  
 8080 
  
 weight 
 : 
  
 100 
 EOF

Using your private client, send a continuous curl request to the internal- http Gateway.

  while 
  
true ; 
  
 do 
  
curl  
-H  
 "host: store.example.internal" 
  
-s  
 VIP 
  
 | 
  
grep  
 "cluster_name" 
 ; 
  
sleep  
 1 
 ; 
  
 done

The output will be similar to this, indicating that all traffic is now going to gke-west-2 .

 "cluster_name": "gke-west-2",
"cluster_name": "gke-west-2",
"cluster_name": "gke-west-2",
"cluster_name": "gke-west-2",
...

This final step completes a full blue-green application migration from one GKE cluster to another GKE cluster.

Deploy capacity-based load balancing

The exercise in this section demonstrates global load balancing and Service capacity concepts by deploying an application across two GKE clusters in different regions. Generated traffic is sent at various request per second (RPS) levels to show how traffic is load balanced across clusters and regions.

The following diagram shows the topology that you will deploy and how traffic overflows between clusters and regions when traffic has exceeded Service capacity:

Traffic overflowing from one cluster to another

To learn more about traffic management, see GKE traffic management .

Prepare your environment

Follow Enabling multi-cluster Gateways to prepare your environment.

Confirm that the GatewayClass resources are installed on the config cluster:

 kubectl  
get  
gatewayclasses  
--context = 
gke-west-1

The output is similar to the following:

 NAME                                  CONTROLLER                  ACCEPTED   AGE
gke-l7-global-external-managed        networking.gke.io/gateway   True       16h
gke-l7-global-external-managed-mc     networking.gke.io/gateway   True       14h
gke-l7-gxlb                           networking.gke.io/gateway   True       16h
gke-l7-gxlb-mc                        networking.gke.io/gateway   True       14h
gke-l7-regional-external-managed      networking.gke.io/gateway   True       16h
gke-l7-regional-external-managed-mc   networking.gke.io/gateway   True       14h
gke-l7-rilb                           networking.gke.io/gateway   True       16h
gke-l7-rilb-mc                        networking.gke.io/gateway   True       14h

Deploy an application

Deploy the sample web application server to both clusters:

 kubectl  
apply  
--context  
gke-west-1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/docs/store-traffic-deploy.yaml
kubectl  
apply  
--context  
gke-east-1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/docs/store-traffic-deploy.yaml

The output is similar to the following:

 namespace/store created
deployment.apps/store created

Deploy a Service, Gateway, and HTTPRoute

Apply the following Service manifest to both gke-west-1 and gke-east-1 clusters:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Service 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 traffic-test 
  
 annotations 
 : 
  
 networking.gke.io/max-rate-per-endpoint 
 : 
  
 "10" 
 spec 
 : 
  
 ports 
 : 
  
 - 
  
 port 
 : 
  
 8080 
  
 targetPort 
 : 
  
 8080 
  
 name 
 : 
  
 http 
  
 selector 
 : 
  
 app 
 : 
  
 store 
  
 type 
 : 
  
 ClusterIP 
 --- 
 kind 
 : 
  
 ServiceExport 
 apiVersion 
 : 
  
 net.gke.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 traffic-test 
 EOF

  cat << EOF | kubectl apply --context gke-east-1 -f - 
 apiVersion 
 : 
  
 v1 
 kind 
 : 
  
 Service 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 traffic-test 
  
 annotations 
 : 
  
 networking.gke.io/max-rate-per-endpoint 
 : 
  
 "10" 
 spec 
 : 
  
 ports 
 : 
  
 - 
  
 port 
 : 
  
 8080 
  
 targetPort 
 : 
  
 8080 
  
 name 
 : 
  
 http 
  
 selector 
 : 
  
 app 
 : 
  
 store 
  
 type 
 : 
  
 ClusterIP 
 --- 
 kind 
 : 
  
 ServiceExport 
 apiVersion 
 : 
  
 net.gke.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 traffic-test 
 EOF

The Service is annotated with max-rate-per-endpoint set to 10 requests per seconds. With 2 replicas per cluster, each Service has 20 RPS of capacity per cluster.

For more information on how to choose a Service capacity level for your Service, see Determine your Service's capacity .

Apply the following Gateway manifest to the config cluster, gke-west-1 in this example:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 Gateway 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 traffic-test 
 spec 
 : 
  
 gatewayClassName 
 : 
  
 gke-l7-global-external-managed-mc 
  
 listeners 
 : 
  
 - 
  
 name 
 : 
  
 http 
  
 protocol 
 : 
  
 HTTP 
  
 port 
 : 
  
 80 
  
 allowedRoutes 
 : 
  
 kinds 
 : 
  
 - 
  
 kind 
 : 
  
 HTTPRoute 
 EOF

The manifest describes an external, global, multi-cluster Gateway that deploys an external Application Load Balancer with a publicly accessible IP address.

Apply the following HTTPRoute manifest to the config cluster, gke-west-1 in this example:

  cat << EOF | kubectl apply --context gke-west-1 -f - 
 kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1beta1 
 metadata 
 : 
  
 name 
 : 
  
 store 
  
 namespace 
 : 
  
 traffic-test 
  
 labels 
 : 
  
 gateway 
 : 
  
 store 
 spec 
 : 
  
 parentRefs 
 : 
  
 - 
  
 kind 
 : 
  
 Gateway 
  
 namespace 
 : 
  
 traffic-test 
  
 name 
 : 
  
 store 
  
 rules 
 : 
  
 - 
  
 backendRefs 
 : 
  
 - 
  
 name 
 : 
  
 store 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 port 
 : 
  
 8080 
 EOF

The manifest describes an HTTPRoute that configures the Gateway with a routing rule that directs all traffic to the store ServiceImport. The store ServiceImport groups the store Service Pods across both clusters and allows them to be addressed by the load balancer as a single Service.

You can check the Gateway's events after a few minutes to see if it has finished deploying:

 kubectl  
describe  
gateway  
store  
-n  
traffic-test  
--context  
gke-west-1

The output is similar to the following:

 ...
Status:
  Addresses:
    Type:   IPAddress
    Value: 34.102.159.147Conditions:
    Last Transition Time:  2023-10-12T21:40:59Z
    Message:               The OSS Gateway API has deprecated this condition, do not depend on it.
    Observed Generation:   1
    Reason:                Scheduled
    Status:                True
    Type:                  Scheduled
    Last Transition Time:  2023-10-12T21:40:59Z
    Message:
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Last Transition Time:  2023-10-12T21:40:59Z
    Message:
    Observed Generation:   1
    Reason:                Programmed
    Status:                True
    Type:                  Programmed
    Last Transition Time:  2023-10-12T21:40:59Z
    Message:               The OSS Gateway API has altered the "Ready" condition semantics and reservedit for future use.  GKE Gateway will stop emitting it in a future update, use "Programmed" instead.
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Listeners:
    Attached Routes:  1
    Conditions:
      Last Transition Time:  2023-10-12T21:40:59Z
      Message:
      Observed Generation:   1
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
      Last Transition Time:  2023-10-12T21:40:59Z
      Message:               The OSS Gateway API has altered the "Ready" condition semantics and reservedit for future use.  GKE Gateway will stop emitting it in a future update, use "Programmed" instead.
      Observed Generation:   1
      Reason:                Ready
      Status:                True
      Type:                  Ready
    Name:                    http
    Supported Kinds:
      Group:  gateway.networking.k8s.io
      Kind:   HTTPRoute
Events:
  Type    Reason  Age                  From                   Message
  ----    ------  ----                 ----                   -------
  Normal  ADD     12m                  mc-gateway-controller  traffic-test/store
  Normal  SYNC    6m43s                mc-gateway-controller  traffic-test/store
  Normal  UPDATE  5m40s (x4 over 12m)  mc-gateway-controller  traffic-test/store
  Normal  SYNC    118s (x6 over 10m)   mc-gateway-controller  SYNC on traffic-test/store was a success

This output shows that the Gateway has deployed successfully. It might still take a few minutes for traffic to start passing after the Gateway has deployed. Take note of the IP address in this output, as it is used in a following step.

Confirm traffic

Confirm that traffic is passing to the application by testing the Gateway IP address with a curl command:

 curl  
 GATEWAY_IP_ADDRESS

The output is similar to the following:

 {
  "cluster_name": "gke-west-1",
  "host_header": "34.117.182.69",
  "pod_name": "store-54785664b5-mxstv",
  "pod_name_emoji": "👳🏿",
  "project_id": "project",
  "timestamp": "2021-11-01T14:06:38",
  "zone": "us-west1-a"
}

This output shows the Pod metadata, which indicates the region where the request was served from.

Verify traffic using load testing

To verify the load balancer is working, you can deploy a traffic generator in your gke-west-1 cluster. The traffic generator generates traffic at different levels of load to demonstrate the capacity and overflow capabilities of the load balancer. The following steps demonstrate three levels of load:

10 RPS, which is under the capacity for the store Service in gke-west-1 .
30 RPS, which is over capacity for the gke-west-1 store Service and causes traffic overflow to gke-east-1 .
60 RPS, which is over capacity for the Services in both clusters.

Configure dashboard

Get the name of the underying URLmap for your Gateway:

 kubectl  
get  
gateway  
store  
-n  
traffic-test  
--context = 
gke-west-1  
-o = 
 jsonpath 
 = 
 "{.metadata.annotations.networking\.gke\.io/url-maps}"

The output is similar to the following:

 /projects/ PROJECT_NUMBER 
/global/urlMaps/gkemcg1-traffic-test-store-armvfyupay1t

In the Google Cloud console, go to the Metrics explorerpage.

Go to Metrics explorer
Under Select a metric, click CODE: MQL.

Enter the following query to observe traffic metrics for the store Service across your two clusters:

 fetch  
https_lb_rule | 
  
metric  
 'loadbalancing.googleapis.com/https/backend_request_count' 
 | 
  
filter  
 ( 
resource.url_map_name  
 == 
  
 ' GATEWAY_URL_MAP 
' 
 ) 
 | 
  
align  
rate ( 
1m ) 
 | 
  
every  
1m | 
  
group_by  
 [ 
resource.backend_scope ] 
,  
 [ 
value_backend_request_count_aggregate:  
aggregate ( 
value.backend_request_count )]

Replace GATEWAY_URL_MAP with the URLmap name from the previous step.

Click Run query. Wait at least 5 minutes after deploying the load generator in the next section for the metrics to display in the chart.

Test with 10 RPS

Deploy a Pod to your gke-west-1 cluster:
```
 kubectl  
run  
--context  
gke-west-1  
-i  
--tty  
--rm  
loadgen  
 \ 
  
--image = 
cyrilbkr/httperf  
 \ 
  
--restart = 
Never  
 \ 
  
--  
/bin/sh  
-c  
 'httperf  \ 
 --server= GATEWAY_IP_ADDRESS 
\ 
 --hog --uri="/zone" --port 80  --wsess=100000,1,1 --rate 10' 
 
```
Replace GATEWAY_IP_ADDRESS with the Gateway IP address from the previous step.

The output is similar to the following, indicating that the traffic generator is sending traffic:
```
 If you don't see a command prompt, try pressing enter. 
```
The load generator continuously sends 10 RPS to the Gateway. Even though traffic is coming from inside a Google Cloud region, the load balancer treats it as client traffic coming from the US West Coast. To simulate a realistic client diversity, the load generator sends each HTTP request as a new TCP connection, which means traffic is distributed across backend Pods more evenly.

The generator takes up to 5 minutes to generate traffic for the dashboard.
View your Metrics explorer dashboard. Two lines appear, indiciating how much traffic is load balanced to each of the clusters:

You should see that us-west1-a is receiving approximately 10 RPS of traffic while us-east1-b is not receiving any traffic. Because the traffic generator is running in us-west1 , all traffic is sent to the Service in the gke-west-1 cluster.

Stop the load generator using Ctrl+C, then delete the pod:

 kubectl  
delete  
pod  
loadgen  
--context  
gke-west-1

Test with 30 RPS

Deploy the load generator again, but configured to send 30 RPS:

 kubectl  
run  
--context  
gke-west-1  
-i  
--tty  
--rm  
loadgen  
 \ 
  
--image = 
cyrilbkr/httperf  
 \ 
  
--restart = 
Never  
 \ 
  
--  
/bin/sh  
-c  
 'httperf  \ 
 --server= GATEWAY_IP_ADDRESS 
\ 
 --hog --uri="/zone" --port 80  --wsess=100000,1,1 --rate 30'

The generator takes up to 5 minutes to generate traffic for the dashboard.

View your Cloud Ops dashboard.

You should see that approximately 20 RPS is being sent to us-west1-a and 10 RPS to us-east1-b . This indicates that the Service in gke-west-1 is fully utilized and is overflowing 10 RPS of traffic to the Service in gke-east-1 .

Stop the load generator using Ctrl+C, then delete the Pod:

 kubectl  
delete  
pod  
loadgen  
--context  
gke-west-1

Test with 60 RPS

Deploy the load generator configured to send 60 RPS:

 kubectl  
run  
--context  
gke-west-1  
-i  
--tty  
--rm  
loadgen  
 \ 
  
--image = 
cyrilbkr/httperf  
 \ 
  
--restart = 
Never  
 \ 
  
--  
/bin/sh  
-c  
 'httperf  \ 
 --server= GATEWAY_IP_ADDRESS 
\ 
 --hog --uri="/zone" --port 80  --wsess=100000,1,1 --rate 60'

Wait 5 minutes and view your Cloud Ops dashboard. It should now show that both clusters are receiving roughly 30 RPS. Since all Services are overutilized globally, there is no traffic spillover and Services absorb all the traffic they can.

Stop the load generator using Ctrl+C, then delete the Pod:

 kubectl  
delete  
pod  
loadgen  
--context  
gke-west-1

Clean up

After completing the exercises on this page, follow these steps to remove resources and prevent unwanted charges incurring on your account:

Delete the clusters .
Unregister the clusters from the fleet if they don't need to be registered for another purpose.

Disable the multiclusterservicediscovery feature:

 gcloud  
container  
fleet  
multi-cluster-services  
disable

Disable Multi Cluster Ingress:

 gcloud  
container  
fleet  
ingress  
disable

Disable the APIs:

 gcloud  
services  
disable  
 \ 
  
multiclusterservicediscovery.googleapis.com  
 \ 
  
multiclusteringress.googleapis.com  
 \ 
  
trafficdirector.googleapis.com  
 \ 
  
--project = 
 PROJECT_ID

Use multi-cluster Gateway with Shared VPC

A multi-cluster Gateway can also be deployed in a Shared VPC environment, with different topologies, depending on the use case.

The following table describes the supported multi-cluster Gateway topologies within a Shared VPC environment:

Scenario	Fleet host project	Config cluster	Workload clusters
1	Shared VPC host project	Shared VPC host project	Shared VPC host project
2	Shared VPC service project	Shared VPC service project (Same as fleet service project)	Shared VPC service project (Same as fleet service project)

To create multi-cluster Gateways in a Shared VPC environment, use the following steps:

Follow the steps to set up your multi-cluster Services with Shared VPC
Create your services and export them to the config cluster
If you plan to use an multi-cluster internal Gateway, create a proxy-only subnet
Create your multi-cluster external or internal Gateway and HTTPRoutes

Once you are done with these steps, you can validate your deployment, depending on your topology.

Troubleshooting

Proxy-only subnet for internal Gateway does not exist

If the following event appears on your internal Gateway, a proxy-only subnet does not exist for that region. To resolve this issue, deploy a proxy-only subnet.

 generic::invalid_argument: error ensuring load balancer: Insert: Invalid value for field 'resource.target': 'regions/us-west1/targetHttpProxies/gkegw-x5vt-default-internal-http-2jzr7e3xclhj'. A reserved and active subnetwork is required in the same region and VPC as the forwarding rule.

No healthy upstream

Symptom:

The following issue might occur when you create a Gateway but cannot access the backend services (503 response code):

 no healthy upstream

Reason:

This error message indicates that the health check prober cannot find healthy backend services. It is possible that your backend services are healthy but you might need to customize the health checks.

Workaround:

To resolve this issue, customize your health check based on your application's requirements (for example, /health ) using a HealthCheckPolicy .

What's next

Learn more about the Gateway controller .

Deploying multi-cluster Gateways Stay organized with collections Save and categorize content based on your preferences.

Multi-cluster Gateways

Multi-cluster, multi-region, external Gateway

Deploying the demo application

Multi-cluster Services

Exporting Services

Deploying the Gateway and HTTPRoute

Validating deployment

Deploy an internal multi-cluster Gateway across regions

Before you begin

Configure an internal multi-region Gateway

Attach HTTPRoutes to the Gateway

Verify the status of the Gateway and Route

Confirm traffic

Use static IP Addresses

Special considerations for non-default subnets

Blue-green, multi-cluster routing with Gateway

Prerequisites

Configuring a proxy-only subnet

Deploying the Gateway

Header-based canary

Traffic mirror

Traffic split

Traffic cut over

Deploy capacity-based load balancing

Prepare your environment

Deploy an application

Deploy a Service, Gateway, and HTTPRoute

Confirm traffic

Verify traffic using load testing

Configure dashboard

Test with 10 RPS

Test with 30 RPS

Test with 60 RPS

Clean up

Use multi-cluster Gateway with Shared VPC

Troubleshooting

Proxy-only subnet for internal Gateway does not exist

No healthy upstream

What's next

Deploying multi-cluster Gateways