Console
    Start free

    Deploy an internal multi-cluster Gateway

    This document guides you through a practical example to deploy an internal multi-cluster Gateway to route traffic within your VPC network to an application that runs in two different GKE clusters.

    Multi-cluster Gateways provide a powerful way to manage traffic for services deployed across multiple GKE clusters. By using Google's global load-balancing infrastructure, you can create a single entry point for your applications, which simplifies management and improves reliability.

    In this tutorial, you use a sample store application to simulate a real-world scenario where an online shopping service is owned and operated by separate teams and deployed across a fleet of shared GKE clusters.

    This example shows you how to set up path-based routing to direct traffic to different clusters.

    Before you begin

    Multi-cluster Gateways require some environmental preparation before they can be deployed. Before you proceed, follow the steps in Prepare your environment for multi-cluster Gateways :

    1. Deploy GKE clusters.

    2. Register your clusters to a fleet (if they aren't already).

    3. Enable the multi-cluster Service and multi-cluster Gateway controllers.

    Finally, review the GKE Gateway controller limitations and known issues before you use the controller in your environment.

    Deploy an internal multi-cluster Gateway across regions

    You can deploy multi-cluster Gateways that provide internal Layer 7 load balancing across GKE clusters in multiple regions. These Gateways use the gke-l7-cross-regional-internal-managed-mc GatewayClass. This GatewayClass provisions a cross-region internal Application Load Balancer that's managed by Google Cloud and that enables internal VIPs that clients within your VPC network can access. These Gateways can be exposed by frontends in the regions of your choice, simply by using the Gateway to request addresses in those regions. The internal VIP can be a single IP address, or they can be IP addresses in multiple regions, with one IP address per region that's specified in the Gateway. Traffic is directed to the closest healthy backend GKE cluster that can serve the request.

    Prerequisites

    1. Set up your project and shell by configuring your gcloud environment with your project ID:

        export 
  
 PROJECT_ID 
 = 
 " YOUR_PROJECT_ID 
" 
gcloud  
config  
 set 
  
project  
 ${ 
 PROJECT_ID 
 }

    2. Create GKE clusters in different regions.

      This example uses two clusters, gke-west-1 in us-west1 and gke-east-1 in us-east1 . Ensure the Gateway API is enabled ( --gateway-api=standard ) and clusters are registered to a fleet.

       gcloud  
container  
clusters  
create  
gke-west-1  
 \ 
  
--location = 
us-west1-a  
 \ 
  
--workload-pool = 
 ${ 
 PROJECT_ID 
 } 
.svc.id.goog  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
  
 \ 
  
--enable-fleet  
 \ 
  
--gateway-api = 
standard

gcloud  
container  
clusters  
create  
gke-east-1  
 \ 
  
--location = 
us-east1-c  
 \ 
  
--workload-pool = 
 ${ 
 PROJECT_ID 
 } 
.svc.id.goog  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
  
 \ 
  
--enable-fleet  
 \ 
  
--gateway-api = 
standard

      Rename contexts for easier access:

       gcloud  
container  
clusters  
get-credentials  
gke-west-1  
 \ 
  
--location = 
us-west1-a  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
gcloud  
container  
clusters  
get-credentials  
gke-east-1  
 \ 
  
--location = 
us-east1-c  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
kubectl  
config  
rename-context  
gke_ ${ 
 PROJECT_ID 
 } 
_us-west1-a_gke-west-1  
gke-west1
kubectl  
config  
rename-context  
gke_ ${ 
 PROJECT_ID 
 } 
_us-east1-c_gke-east-1  
gke-east1

    3. Enable Multi-Cluster Services (MCS) and Multi-Cluster Ingress (MCI/Gateway):

       gcloud  
container  
fleet  
multi-cluster-services  
 enable 
  
--project = 
 ${ 
 PROJECT_ID 
 } 
 # Set the config membership to one of your clusters (e.g., gke-west-1) 
 # This cluster will be the source of truth for multi-cluster Gateway and Route resources. 
gcloud  
container  
fleet  
ingress  
 enable 
  
 \ 
  
--config-membership = 
projects/ ${ 
 PROJECT_ID 
 } 
/locations/us-west1/memberships/gke-west-1  
 \ 
  
--project = 
 ${ 
 PROJECT_ID 
 }

    4. Configure proxy-only subnets. A proxy-only subnet is required in each region where your GKE clusters are located and where the load balancer will operate. Cross-region internal Application Load Balancers require the purpose of this subnet to be set to GLOBAL_MANAGED_PROXY .

        # Proxy-only subnet for us-west1 
gcloud  
compute  
networks  
subnets  
create  
us-west1-proxy-only-subnet  
 \ 
  
--purpose = 
GLOBAL_MANAGED_PROXY  
 \ 
  
--role = 
ACTIVE  
 \ 
  
--region = 
us-west1  
 \ 
  
--network = 
default  
 \ 
  
--range = 
 10 
.129.0.0/23  
 # Choose an appropriate unused CIDR range 
 # Proxy-only subnet for us-east1 
gcloud  
compute  
networks  
subnets  
create  
us-east1-proxy-only-subnet  
 \ 
  
--purpose = 
GLOBAL_MANAGED_PROXY  
 \ 
  
--role = 
ACTIVE  
 \ 
  
--region = 
us-east1  
 \ 
  
--network = 
default  
 \ 
  
--range = 
 10 
.130.0.0/23  
 # Choose an appropriate unused CIDR range

      If you're not using the default network, replace default with the name of your VPC network. Ensure that the CIDR ranges are unique and don't overlap.

    5. Deploy your demo applications, such as store , to both clusters. The example store.yaml file from gke-networking-recipes creates a store namespace and a deployment.

       kubectl  
apply  
--context  
gke-west1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml
kubectl  
apply  
--context  
gke-east1  
-f  
https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/main/gateway/gke-gateway-controller/multi-cluster-gateway/store.yaml

    6. Export Services from each cluster by creating Kubernetes Service resources and ServiceExport resources in each cluster, which makes the services discoverable across the fleet. The following example exports a generic store service and region-specific services ( store-west-1 , store-east-1 ) from each cluster, all within the store namespace.

      Apply to gke-west1 :

       cat << 
EOF  
 | 
  
kubectl  
apply  
--context  
gke-west1  
-f  
-
apiVersion:  
v1
kind:  
Service
metadata:  
name:  
store  
namespace:  
store
spec:  
selector:  
app:  
store  
ports:  
-  
port:  
 8080 
  
targetPort:  
 8080 
---
kind:  
ServiceExport
apiVersion:  
net.gke.io/v1
metadata:  
name:  
store  
namespace:  
store
---
apiVersion:  
v1
kind:  
Service
metadata:  
name:  
store-west-1  
 # Specific to this cluster 
  
namespace:  
store
spec:  
selector:  
app:  
store  
ports:  
-  
port:  
 8080 
  
targetPort:  
 8080 
---
kind:  
ServiceExport
apiVersion:  
net.gke.io/v1
metadata:  
name:  
store-west-1  
 # Exporting the region-specific service 
  
namespace:  
store
EOF

      Apply to gke-east1 :

       cat << 
EOF  
 | 
  
kubectl  
apply  
--context  
gke-east1  
-f  
-
apiVersion:  
v1
kind:  
Service
metadata:  
name:  
store  
namespace:  
store
spec:  
selector:  
app:  
store  
ports:  
-  
port:  
 8080 
  
targetPort:  
 8080 
---
kind:  
ServiceExport
apiVersion:  
net.gke.io/v1
metadata:  
name:  
store  
namespace:  
store
---
apiVersion:  
v1
kind:  
Service
metadata:  
name:  
store-east-1  
 # Specific to this cluster 
  
namespace:  
store
spec:  
selector:  
app:  
store  
ports:  
-  
port:  
 8080 
  
targetPort:  
 8080 
---
kind:  
ServiceExport
apiVersion:  
net.gke.io/v1
metadata:  
name:  
store-east-1  
 # Exporting the region-specific service 
  
namespace:  
store
EOF

    7. Check ServiceImports: Verify that ServiceImport resources are created in each cluster within the store namespace. It might take a few minutes for them to be created. bash kubectl get serviceimports --context gke-west1 -n store kubectl get serviceimports --context gke-east1 -n store You should see store , store-west-1 , and store-east-1 listed (or relevant entries based on propagation).

    Configure an internal multi-region Gateway

    Define a Gateway resource that references the gke-l7-cross-regional-internal-managed-mc GatewayClass. You apply this manifest to your designated config cluster, such as gke-west-1 .

    The spec.addresses field lets you request ephemeral IP addresses in specific regions or use pre-allocated static IP addresses.

    1. To use ephemeral IP addresses, save the following Gateway manifest as cross-regional-gateway.yaml :

        # cross-regional-gateway.yaml 
 kind 
 : 
  
 Gateway 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 internal-cross-region-gateway 
  
 namespace 
 : 
  
 store 
  
 # Namespace for the Gateway resource 
 spec 
 : 
  
 gatewayClassName 
 : 
  
 gke-l7-cross-regional-internal-managed-mc 
  
 addresses 
 : 
  
 # Addresses across regions. Address value is allowed to be empty or matching 
  
 # the region name. 
  
 - 
  
 type 
 : 
  
 networking.gke.io/ephemeral-ipv4-address/us-west1 
  
 value 
 : 
  
 "us-west1" 
  
 - 
  
 type 
 : 
  
 networking.gke.io/ephemeral-ipv4-address/us-east1 
  
 value 
 : 
  
 "us-east1" 
  
 listeners 
 : 
  
 - 
  
 name 
 : 
  
 http 
  
 protocol 
 : 
  
 HTTP 
  
 port 
 : 
  
 80 
  
 allowedRoutes 
 : 
  
 kinds 
 : 
  
 - 
  
 kind 
 : 
  
 HTTPRoute 
  
 # Only allow HTTPRoute to attach

      The following list defines some of the fields in the previous YAML file:

      • metadata.namespace : the namespace where the Gateway resource is created, for example, store .
      • spec.gatewayClassName : the name of the GatewayClass. Must be gke-l7-cross-regional-internal-managed-mc .
      • spec.listeners.allowedRoutes.kinds : the kinds of Route objects that can be attached, for example, HTTPRoute .
      • spec.addresses :
        • type: networking.gke.io/ephemeral-ipv4-address/ REGION : requests an ephemeral IP address.
        • value : specifies the region for the address, for example, "us-west1" or "us-east1" .

    2. Apply the manifest to your config cluster, for example, gke-west1 :

       kubectl  
apply  
--context  
gke-west1  
-f  
cross-regional-gateway.yaml

    Attach HTTPRoutes to the Gateway

    Define HTTPRoute resources to manage traffic routing and apply them to your config cluster.

    1. Save the following HTTPRoute manifest as store-route.yaml :

        # store-route.yaml 
 kind 
 : 
  
 HTTPRoute 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 store-route 
  
 namespace 
 : 
  
 store 
  
 labels 
 : 
  
 gateway 
 : 
  
 cross-regional-internal 
 spec 
 : 
  
 parentRefs 
 : 
  
 - 
  
 name 
 : 
  
 internal-cross-region-gateway 
  
 namespace 
 : 
  
 store 
  
 # Namespace where the Gateway is deployed 
  
 hostnames 
 : 
  
 - 
  
 "store.example.internal" 
  
 # Hostname clients will use 
  
 rules 
 : 
  
 - 
  
 matches 
 : 
  
 # Rule for traffic to /west 
  
 - 
  
 path 
 : 
  
 type 
 : 
  
 PathPrefix 
  
 value 
 : 
  
 /west 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 # Indicates a multi-cluster ServiceImport 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-west-1 
  
 # Targets the ServiceImport for the west cluster 
  
 port 
 : 
  
 8080 
  
 - 
  
 matches 
 : 
  
 # Rule for traffic to /east 
  
 - 
  
 path 
 : 
  
 type 
 : 
  
 PathPrefix 
  
 value 
 : 
  
 /east 
  
 backendRefs 
 : 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store-east-1 
  
 # Targets the ServiceImport for the east cluster 
  
 port 
 : 
  
 8080 
  
 - 
  
 backendRefs 
 : 
  
 # Default rule for other paths (e.g., /) 
  
 - 
  
 group 
 : 
  
 net.gke.io 
  
 kind 
 : 
  
 ServiceImport 
  
 name 
 : 
  
 store 
  
 # Targets the generic 'store' ServiceImport (any region) 
  
 port 
 : 
  
 8080

      The following list defines some of the fields in the previous YAML file:

      • spec.parentRefs : attaches this route to internal-cross-region-gateway in the store namespace.
      • spec.hostnames : represents the hostname that clients use to access the service.
      • spec.rules : defines routing logic. This example uses path-based routing:
        • /west traffic goes to store-west-1 ServiceImport.
        • /east traffic goes to store-east-1 ServiceImport.
        • All other traffic, such as / , goes to the generic store ServiceImport.
      • backendRefs :
        • group: net.gke.io and kind: ServiceImport target multi-cluster services.

    2. Apply the HTTPRoute manifest to your config cluster:

       kubectl  
apply  
--context  
gke-west1  
-f  
store-route.yaml

    Verify the status of the Gateway and Route

    1. Check the Gateway status:

       kubectl  
get  
gateway  
internal-cross-region-gateway  
-n  
store  
-o  
yaml  
--context  
gke-west1

      Look for a condition with type: Programmed and status: "True" . You should see IP addresses assigned in the status.addresses field, corresponding to the regions you specified (e.g., one for us-west1 and one for us-east1`).

    2. Check the HTTPRoute status:

       kubectl  
get  
httproute  
store-route  
-n  
store  
-o  
yaml  
--context  
gke-west1

      Look for a condition in status.parents[].conditions with type: Accepted (or ResolvedRefs ) and status: "True" .

    Confirm traffic

    After you assign the IP addresses to the Gateway, you can test traffic from a client VM that's within your VPC network and in one of the regions, or in a region that can connect to the Gateway IP address.

    1. Retrieve the Gateway IP addresses.

      The following command attempts to parse the JSON output. You might need to adjust the jsonpath based on the exact structure.

       kubectl  
get  
gateway  
cross-region-gateway  
-n  
store  
--context  
gke-west1  
-o = 
 jsonpath 
 = 
 "{.status.addresses[*].value}" 
.

      The output of this command should include the VIPs, such as VIP1_WEST , or VIP2_EAST .

    2. Send test requests: From a client VM in your VPC:

        # Assuming VIP_WEST is an IP in us-west1 and VIP_EAST is an IP in us-east1 
 # Traffic to /west should ideally be served by gke-west-1 
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_WEST/west
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_EAST/west  
 # Still targets store-west-1 due to path 
 # Traffic to /east should ideally be served by gke-east-1 
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_WEST/east  
 # Still targets store-east-1 due to path 
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_EAST/east # Traffic to / (default) could be served by either cluster 
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_WEST/
curl  
-H  
 "host: store.example.internal" 
  
http://VIP_EAST/

      The response should include details from the store application that indicate which backend pod served the request, such as cluster_name or zone .

    Use static IP Addresses

    Instead of ephemeral IP addresses, you can use pre-allocated static internal IP addresses.

    1. Create static IP addresses in the regions that you want to use:

       gcloud  
compute  
addresses  
create  
cross-region-gw-ip-west  
--region  
us-west1  
--subnet  
default  
--project = 
 ${ 
 PROJECT_ID 
 } 
gcloud  
compute  
addresses  
create  
cross-region-gw-ip-east  
--region  
us-east1  
--subnet  
default  
--project = 
 ${ 
 PROJECT_ID 
 }

      If you're not using the default subnet, replace default with the name of the subnet that has the IP address you want to allocate. These subnets are regular subnets, not the proxy-only subnets.

    2. Update the Gateway manifest by modifying the spec.addresses section in your cross-regional-gateway.yaml file:

        # cross-regional-gateway-static-ip.yaml 
 kind 
 : 
  
 Gateway 
 apiVersion 
 : 
  
 gateway.networking.k8s.io/v1 
 metadata 
 : 
  
 name 
 : 
  
 internal-cross-region-gateway 
  
 # Or a new name if deploying alongside 
  
 namespace 
 : 
  
 store 
 spec 
 : 
  
 gatewayClassName 
 : 
  
 gke-l7-cross-regional-internal-managed-mc 
  
 addresses 
 : 
  
 - 
  
 type 
 : 
  
 networking.gke.io/named-address-with-region 
  
 # Use for named static IP 
  
 value 
 : 
  
 "regions/us-west1/addresses/cross-region-gw-ip-west" 
  
 - 
  
 type 
 : 
  
 networking.gke.io/named-address-with-region 
  
 value 
 : 
  
 "regions/us-east1/addresses/cross-region-gw-ip-east" 
  
 listeners 
 : 
  
 - 
  
 name 
 : 
  
 http 
  
 protocol 
 : 
  
 HTTP 
  
 port 
 : 
  
 80 
  
 allowedRoutes 
 : 
  
 kinds 
 : 
  
 - 
  
 kind 
 : 
  
 HTTPRoute

    3. Apply the updated Gateway manifest.

       kubectl  
apply  
--context  
gke-west1  
-f  
cross-regional-gateway.yaml

    Special considerations for non-default subnets

    Be aware of the following considerations when you use non-default subnets:

    • Same VPC network:all user-created resources—such as static IP addresses, proxy-only subnets, and GKE clusters—must reside within the same VPC network.

    • Address subnet:when you create static IP addresses for the Gateway, they are allocated from regular subnets in the specified regions.

    • Cluster subnet naming:Each region must have a subnet that has the same name as the subnet that the MCG config cluster resides in.

      • For example, if your gke-west-1 config cluster is in projects/YOUR_PROJECT/regions/us-west1/subnetworks/my-custom-subnet , then the regions you are requesting addresses for must also have the my-custom-subnet subnet. If you request addresses in the us-east1 and us-centra1 regions, then a subnet named my-custom-subnet must also exist in those regions.

    Clean up

    After completing the exercises on this document, follow these steps to remove resources and prevent unwanted charges incurring on your account:

    1. Delete the clusters .

    2. Unregister the clusters from the fleet if they don't need to be registered for another purpose.

    3. Disable the multiclusterservicediscovery feature:

       gcloud  
container  
fleet  
multi-cluster-services  
disable

    4. Disable Multi Cluster Ingress:

       gcloud  
container  
fleet  
ingress  
disable

    5. Disable the APIs:

       gcloud  
services  
disable  
 \ 
  
multiclusterservicediscovery.googleapis.com  
 \ 
  
multiclusteringress.googleapis.com  
 \ 
  
trafficdirector.googleapis.com  
 \ 
  
--project = 
 PROJECT_ID

    Troubleshooting

    Proxy-only subnet for internal Gateway does not exist

    If the following event appears on your internal Gateway, a proxy-only subnet does not exist for that region. To resolve this issue, deploy a proxy-only subnet.

     generic::invalid_argument: error ensuring load balancer: Insert: Invalid value for field 'resource.target': 'regions/us-west1/targetHttpProxies/gkegw-x5vt-default-internal-http-2jzr7e3xclhj'. A reserved and active subnetwork is required in the same region and VPC as the forwarding rule.

    No healthy upstream

    Symptom:

    The following issue might occur when you create a Gateway but cannot access the backend services (503 response code):

     no healthy upstream

    Reason:

    This error message indicates that the health check prober cannot find healthy backend services. It is possible that your backend services are healthy but you might need to customize the health checks.

    Workaround:

    To resolve this issue, customize your health check based on your application's requirements (for example, /health ) using a HealthCheckPolicy .

    What's next
    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: