Create an internal load balancer across VPC networks

Autopilot Standard

This page explains how to create an internal passthrough Network Load Balancer on Google Kubernetes Engine (GKE) across VPC networks.

Before reading this page, ensure that you're familiar with the following concepts:

Before you begin

Before you start, make sure that you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update .
Note: For existing gcloud CLI installations, make sure to set the compute/region property . If you use primarily zonal clusters, set the compute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location . You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Create an internal load balancer with Private Service Connect

As a service producer, you can use service attachments to make your services available to service consumers in other VPC networks using Private Service Connect . You can create, manage, and delete service attachments using a ServiceAttachment custom resource.

Requirements and limitations

Limitations for Private Service Connect apply.
You can create a service attachment in GKE versions 1.21.4-gke.300 and later.
Your cluster must have the HttpLoadBalancing add-on enabled. New GKE clusters have the HttpLoadBalancing add-on enabled by default.
You cannot use the same subnet in multiple service attachment configurations.
You must create a GKE service that uses an internal passthrough Network Load Balancer.
You cannot specify a subnet in a different project (Shared VPC) for GKE versions earlier than 1.22.4-gke.100. For Shared VPC, ensure all requirements for Shared VPC are met.
After you create a service attachment, you can't update the internal load balancer. To change your load balancer configurations, you must delete and recreate the service attachment.

Create a `ServiceAttachment`

Create a subnet.

You must create a new subnet for each ServiceAttachment .
```
 gcloud  
beta  
compute  
networks  
subnets  
create  
 SUBNET_NAME 
  
 \ 
  
--project  
 PROJECT_ID 
  
 \ 
  
--network  
 NETWORK_NAME 
  
 \ 
  
--region  
 REGION 
  
 \ 
  
--range  
 SUBNET_RANGE 
  
 \ 
  
--purpose  
PRIVATE_SERVICE_CONNECT 
```
Replace the following:
- SUBNET_NAME : the name of the new subnet. In GKE versions 1.22.4-gke.100 and later, you can specify a subnet in a different project by using the fully qualified resource URL for this field. You can get the fully qualified resource URL using the command gcloud compute networks subnets describe .
- PROJECT_ID : the ID of your Google Cloud project.
- NETWORK_NAME : the name of the VPC network for the subnet.
- REGION : the region for the new subnet. You must use the same region as the service that you create.
- SUBNET_RANGE : the IP address range to use for the subnet.

Deploy a workload.

The following manifest describes a Deployment that runs a sample web application container image. Save the manifest as my-deployment.yaml :

  apiVersion 
 : 
  
 apps/v1 
 kind 
 : 
  
 Deployment 
 metadata 
 : 
  
 name 
 : 
  
 psc-ilb 
 spec 
 : 
  
 replicas 
 : 
  
 3 
  
 selector 
 : 
  
 matchLabels 
 : 
  
 app 
 : 
  
 psc-ilb 
  
 template 
 : 
  
 metadata 
 : 
  
 labels 
 : 
  
 app 
 : 
  
 psc-ilb 
  
 spec 
 : 
  
 containers 
 : 
  
 - 
  
 name 
 : 
  
 whereami 
  
 image 
 : 
  
 us-docker.pkg.dev/google-samples/containers/gke/whereami:v1 
  
 ports 
 : 
  
 - 
  
 name 
 : 
  
 http 
  
 containerPort 
 : 
  
 8080 
  
 readinessProbe 
 : 
  
 httpGet 
 : 
  
 path 
 : 
  
 /healthz 
  
 port 
 : 
  
 8080 
  
 scheme 
 : 
  
 HTTP 
  
 initialDelaySeconds 
 : 
  
 5 
  
 timeoutSeconds 
 : 
  
 1

Apply the manifest to your cluster:

 kubectl  
apply  
-f  
my-deployment.yaml

Create a service. The following manifest describes a service that creates an internal passthrough Network Load Balancer on TCP port 8080. Save the manifest as my-service.yaml :

   
 apiVersion 
 : 
  
 v1 
  
 kind 
 : 
  
 Service 
  
 metadata 
 : 
  
 name 
 : 
  
  SERVICE_NAME 
 
  
 annotations 
 : 
  
 networking.gke.io/load-balancer-type 
 : 
  
 "Internal" 
  
 spec 
 : 
  
 type 
 : 
  
 LoadBalancer 
  
 selector 
 : 
  
 app 
 : 
  
 psc-ilb 
  
 ports 
 : 
  
 - 
  
 port 
 : 
  
 80 
  
 targetPort 
 : 
  
 8080 
  
 protocol 
 : 
  
 TCP

Replace the following:

SERVICE_NAME : the name of the new service.

Apply the manifest to your cluster:

 kubectl  
apply  
-f  
my-service.yaml

Create ServiceAttachment .

The following manifest describes a ServiceAttachment that exposes the service that you created to service consumers. Save the manifest as my-psc.yaml :
```
  apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 ServiceAttachment 
 metadata 
 : 
  
 name 
 : 
  
  SERVICE_ATTACHMENT_NAME 
 
  
 namespace 
 : 
  
 default 
 spec 
 : 
  
 connectionPreference 
 : 
  
 ACCEPT_AUTOMATIC 
  
 natSubnets 
 : 
  
 - 
  
  SUBNET_NAME 
 
  
 proxyProtocol 
 : 
  
 false 
  
 resourceRef 
 : 
  
 kind 
 : 
  
 Service 
  
 name 
 : 
  
  SERVICE_NAME 
 
 
```
Note: For GKE versions earlier than 1.23.3-gke.900, use networking.gke.io/v1beta1 instead of networking.gke.io/v1 .

Replace the following:
- SERVICE_ATTACHMENT_NAME : the name of the new service attachment.
- SUBNET_NAME : the name of the new subnet. In GKE versions 1.22.4-gke.100 and later, you can specify a subnet in a different project by using the fully qualified resource URL for this field. You can get the fully qualified resource URL using the command gcloud compute networks subnets describe . For a Shared VPC configuration, use the following format: projects/ HOST_PROJECT_ID /regions/ COMPUTE_REGION /subnetworks/ SUBNET_NAME .
Note: The ServiceAttachment resource will have error events until the GKE controller creates the internal passthrough Network Load Balancer that is defined by the service. After the internal passthrough Network Load Balancer is provisioned, the errors are no longer published.

For more information about the manifest fields, see the service attachment fields .
Apply the manifest to your cluster:
```
 kubectl  
apply  
-f  
my-psc.yaml 
```

Verify that the Private Service Connect controller created the service attachment:

 gcloud  
beta  
compute  
service-attachments  
list

The output shows a service attachment with an automatically generated name:

 NAME        REGION       PRODUCER_FORWARDING_RULE          CONNECTION_PREFERENCE
k8s1-sa-... REGION_NAME  a3fea439c870148bdba5e59c9ea9451a  ACCEPT_AUTOMATIC

View a `ServiceAttachment`

You can view the details of a ServiceAttachment using the following command:

 kubectl  
describe  
serviceattachment  
 SERVICE_ATTACHMENT_NAME

The output is similar to the following:

 Name:        <sa-name>
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  networking.gke.io/v1beta1
Kind:         ServiceAttachment
Metadata:
  ...
Status:
  Forwarding Rule URL:      https://www.googleapis.com/compute/beta/projects/<project>/regions/<region>/forwardingRules/<fr-name>
  Last Modified Timestamp:  2021-07-08T01:32:39Z
  Service Attachment URL:   https://www.googleapis.com/compute/beta/projects/<projects>/regions/<region>/serviceAttachments/<gce-service-attachment-name>
Events:                     <none>

Consume a `ServiceAttachment`

To consume your service from another project, perform the following steps:

Get the URL of the ServiceAttachment :

 kubectl  
get  
serviceattachment  
 SERVICE_ATTACHMENT_NAME 
  
-o = 
 jsonpath 
 = 
 "{.status.serviceAttachmentURL}"

The output is similar to the following:

 serviceAttachmentURL: https://www.googleapis.com/compute/alpha/projects/<project>/region/<region>/serviceAttachments/k8s1-...my-sa

Create a Private Service Connect endpoint using the URL of the ServiceAttachment .

Verify that you can connect to the Service that you deployed in the producer project by using a curl command from a VM in the consumer project:

 curl  
 PSC_IP_ADDRESS

Replace PSC_IP_ADDRESS with the IP address of the forwarding rule in the consumer project.

The output is similar to the following:

 {
  "cluster_name":"cluster",
  "host_header":"10.128.15.200",
  "node_name":"gke-psc-default-pool-be9b6e0e-dvxg.c.gke_project.internal",
  "pod_name":"foo-7bf648dcfd-l5jf8",
  "pod_name_emoji":"👚",
  "project_id":"gke_project",
  "timestamp":"2021-06-29T21:32:03",
  "zone":"ZONE_NAME"
}

Update a `ServiceAttachment`

You can update a ServiceAttachment using the following steps:

Edit the ServiceAttachment manifest in my-psc.yaml :

  apiVersion 
 : 
  
 networking.gke.io/v1 
 kind 
 : 
  
 ServiceAttachment 
 metadata 
 : 
  
 name 
 : 
  
 my-sa 
  
 namespace 
 : 
  
 default 
 spec 
 : 
  
 connectionPreference 
 : 
  
 ACCEPT_AUTOMATIC 
  
 natSubnets 
 : 
  
 - 
  
 my-nat-subnet 
  
 proxyProtocol 
 : 
  
 false 
  
 resourceRef 
 : 
  
 kind 
 : 
  
 Service 
  
 name 
 : 
  
 ilb-service

Apply the manifest to your cluster:
```
 kubectl  
apply  
-f  
my-psc.yaml 
```

Delete a `ServiceAttachment`

You cannot delete an internal passthrough Network Load Balancer that is connected to a service attachment. You must delete the service attachment and GKE Service separately.

Delete the service attachment:
```
 kubectl  
delete  
serviceattachment  
 SERVICE_ATTACHMENT_NAME 
  
--wait = 
 false 
 
```
This command marks the service attachment for deletion, but the resource continues to exist. You can also wait for the deletion to finish by omitting the --wait flag.

Delete the Service:

 kubectl  
delete  
svc  
 SERVICE_NAME

Delete the subnet:

 gcloud  
compute  
networks  
subnets  
delete  
 SUBNET_NAME

`ServiceAttachment` fields

The ServiceAttachment has the following fields:

connectionPreference : the connection preference that determines how customers connect to the service. You can either use automatic project approval using ACCEPT_AUTOMATIC or explicit project approval using ACCEPT_MANUAL . For more information, see Publishing services using Private Service Connect .
natSubnets : a list of subnetwork resource names to use for the service attachment.
proxyProtocol : when set to true, the consumer source IP and Private Service Connect connection ID are available in the requests. This field is optional and defaults to false if not provided.
consumerAllowList : the list of consumer projects that are allowed to connect to the ServiceAttachment . This field can only be used when connectionPreference is ACCEPT_MANUAL . For more information about this field, see Publishing services using Private Service Connect .
- project : the project ID or number for the consumer project.
- connectionLimit : the connection limit for the consumer project. This field is optional.
- forceSendFields : the field names to send to include in API requests. This field is optional.
- nullFields : the field names to include in API requests with a null value. This field is optional.
consumerRejectList : the list of consumer project IDs or numbers that are not allowed to connect to the ServiceAttachment . This field can only be used when connectionPreference is ACCEPT_MANUAL . For more information about this field, see Publishing services using Private Service Connect .
resourceRef : a reference to the Kubernetes resource.
- kind : the type of Kubernetes resource. You must use Service .
- name : the name of the Kubernetes resource that must be in the same namespace as the internal passthrough Network Load Balancer.
Note: After a service attachment is connected to an internal passthrough Network Load Balancer, you cannot delete or update the internal passthrough Network Load Balancer. If you attempt to delete or update the internal passthrough Network Load Balancer, an error message appears. For more information, see Troubleshooting .

Troubleshooting

You can view error messages using the following command:

 kubectl  
get  
events  
-n  
 NAMESPACE

Replace NAMESPACE with the namespace of the internal passthrough Network Load Balancer.

An error message similar to the following occurs if you try to delete an internal passthrough Network Load Balancer that is being used by a service attachment. You must delete the ServiceAttachment before you can delete the internal passthrough Network Load Balancer.

 Error syncing load balancer: failed to ensure load balancer: googleapi:
Error 400: The forwarding_rule resource '<fwd-rule-URL>' is already being used
by '<svc-attachment-URL>', resourceInUseByAnotherResource.

Create an internal load balancer across VPC networks Stay organized with collections Save and categorize content based on your preferences.

Before you begin

Create an internal load balancer with Private Service Connect

Requirements and limitations

Create a ServiceAttachment

View a ServiceAttachment

Consume a ServiceAttachment

Update a ServiceAttachment

Delete a ServiceAttachment

ServiceAttachment fields

Troubleshooting

What's next

Create an internal load balancer across VPC networks

Create a `ServiceAttachment`

View a `ServiceAttachment`

Consume a `ServiceAttachment`

Update a `ServiceAttachment`

Delete a `ServiceAttachment`

`ServiceAttachment` fields