Set up multi-tenant logging

This page explains how to configure multi-tenant logging for Google Kubernetes Engine (GKE) clusters.

It's common for multiple teams to share a single GKE cluster. Sharing a cluster provides multiple advantages including easier service discovery, simplified security, and it means that cluster administrators have fewer clusters to maintain. Individual application teams often have their own separate project , however. This structure, having a main GKE cluster, but separate namespaces for each application team, is called multi-tenancy. The application team's project is called the tenant .

With Google Cloud, GKE cluster admins can create a system where logs for the cluster remain in the main GKE project, and tenant logs are distributed to tenant projects. To configure your logs this way, use the Log Router . The Log Router gives you control over how logs flow within your Google Cloud project and how logs are routed to supported destinations.

To create tenant-specific logs, the cluster admin creates a sink to route log entries to each tenant's project. In each tenant project, individual teams can control how the logs are stored and used, such as monitoring the logs by configuring log-based metrics and log-based alerts.

We recommend that _Default sinks in main GKE projects include an exclusion filter . The exclusion filter prevents tenant logs from being ingested in both the main GKE project and the tenant project.

Prerequisites

Become familiar with the following documents about multi-tenancy and routing:
Ensure that you have an existing tenant project and main project.

Configure multi-tenant logging

You can configure multi-tenant logging by using the Google Cloud CLI or the Google Cloud console.

gcloud

To configure multi-tenant logging for GKE clusters, complete the following steps:

Set the following environment variables:

  export 
  
 TENANT_NAMESPACE 
 = 
 " TENANT_NAMESPACE 
" 
 export 
  
 MAIN_PROJECT 
 = 
 " MAIN_PROJECT_ID 
" 
 export 
  
 TENANT_PROJECT 
 = 
 " TENANT_PROJECT_ID 
"

Replace the following:

TENANT_NAMESPACE : the name for the tenant project namespace
MAIN_PROJECT_ID : the project ID for your main project
TENANT_PROJECT_ID : the project ID for your tenant project

Create a namespace in your multi-tenant cluster:

 kubectl  
create  
namespace  
 $TENANT_NAMESPACE

Create a log sink in the main GKE project:

 gcloud  
logging  
sinks  
create  
gke- $TENANT_NAMESPACE 
-sink  
 \ 
logging.googleapis.com/projects/ $TENANT_PROJECT 
  
 \ 
  
--project = 
 $MAIN_PROJECT 
  
 \ 
  
--log-filter = 
resource.labels.namespace_name = 
 " 
 $TENANT_NAMESPACE 
 " 
  
 \ 
  
--description = 
 "Log sink to 
 $TENANT_PROJECT 
 for 
 $TENANT_NAMESPACE 
 namespace"

This command creates a log sink that sends all logs that are related to the $TENANT_NAMESPACE namespace to the tenant project.

You might need to use a more restrictive --log-filter . For example, if your cluster and tenant have the same namespace, then add a cluster filter.

For more information about these fields, see the gcloud logging sinks create API documentation .

Get the writer identity from the sink in the main project and assign it to an environment variable.

  export 
  
 SERVICE_ACCOUNT 
 = 
 $( 
gcloud  
logging  
sinks  
describe  
gke- $TENANT_NAMESPACE 
-sink  
 \ 
  
--project = 
 $MAIN_PROJECT 
  
 \ 
  
--format = 
 'value(writerIdentity)' 
 )

Grant the Logs Writer ( logging.bucketWriter ) role to the service account used by the sink. The following command grants the main project permissions to write logs to the tenant project:

   
gcloud  
projects  
add-iam-policy-binding  
 $TENANT_PROJECT 
  
 \ 
  
--member = 
 $SERVICE_ACCOUNT 
  
--role = 
 'roles/logging.logWriter' 
  
 \ 
  
--condition = 
 "expression=resource.name.endsWith(\"projects/ 
 $TENANT_PROJECT 
 \"),title=Log writer for 
 $TENANT_NAMESPACE 
 ,description=Grants Logs Writer role to service account 
 $SERVICE_ACCOUNT 
 used by gke- 
 $TENANT_NAMESPACE 
 -sink"

For more information about these fields, see the gcloud projects add-iam-policy-binding API documentation .

Optionally, create an exclusion filter for the sink that routes logs to the _Default bucket of the main project. If you don't create an exclusion filter on the _Default bucket, then the routed logs appear in both the _Default bucket of the main project and the tenant log bucket. To create an exclusion filter, do the following:
```
 gcloud  
logging  
sinks  
update  
_Default  
--project = 
 $MAIN_PROJECT 
  
 \ 
  
--add-exclusion = 
 "name=gke- 
 $TENANT_NAMESPACE 
 -default-exclusion,description=\"Exclusion filter on the _Default bucket for 
 $TENANT_NAMESPACE 
 \",filter=resource.labels.namespace_name=\" 
 $TENANT_NAMESPACE 
 \"" 
 
```
For more information about these fields, see the gcloud logging sinks update API documentation .

Console

To implement multi-tenant logging for GKE, complete the following steps:

Create the log sink in the main project:
1. Use the Google Cloud console project picker to select the main GKE project.
2. In the Google Cloud console, go to the Log Router page:
  
  Go to Log Router
  
  If you use the search bar to find this page, then select the result whose subheading is Logging .
3. In the Log Routerpage, click Create sink.
4. Enter a Nameand Descriptionfor your sink and then click Next.
5. In the Select sink servicemenu, select Other project.
6. In the Sink destinationfield, add the following destination:
```
  logging.googleapis.com/projects/ TENANT_PROJECT_ID 
 
 
```
  Replace TENANT_PROJECT_ID with the project ID of your tenant project.
7. Click Next.
8. In the Build inclusion filteradd the following filter:
```
  resource.labels.namespace_name=" TENANT_NAMESPACE 
" 
 
```
  Replace TENANT_NAMESPACE with the name of your tenant project's namespace.
  
  You might want a more restrictive inclusion filter. For example, when your cluster and tenant have the same namespace, consider adding a clause to include only log entries for a specific cluster.
9. Click Create sink. Your new sink appears in the Log Router Sinkslist.
Copy the sink's writer identity to your clipboard:
1. In the Log Routerpage, locate the log sink.
2. For that sink, click More, and then select View sink details.
3. In the Sink detailspane, locate the Writer identityfield, and then copy the value into your clipboard. Omit serviceAccount: from the copied value.
Note: If the Writer identityfield for your sink lists the value of None, then the sink is routing log entries to a log bucket in the current Google Cloud project. That is, you haven't configured the sink to route log entries to a tenant project. For information about how to route log entries to a log bucket in the same Google Cloud project, see Configure and manage sinks .
In the tenant project, grant the Logs Writer ( roles/logging.logWriter ) role to the service account used by the main project's sink. The main project needs this permission to write logs to the tenant project.
1. In the Google Cloud console, go to the IAM page:
  
  Go to IAM
  
  If you use the search bar to find this page, then select the result whose subheading is IAM & Admin .
2. Click Grant access.
3. In the New principalsfield, add the sink's service account.
4. From the Select a roledrop-down, select Loggingand choose Logs Writer.
5. Click Save.
Optionally, create an exclusion filter in the _Default bucket of the main project to prevent logs that are routed to a tenant project from being written to the _Default log bucket in the main project:
1. In the Google Cloud console, go to the Log Router page:
  
  Go to Log Router
  
  If you use the search bar to find this page, then select the result whose subheading is Logging .
2. Next to the _Default bucket, click Moreand select Edit sink.
3. In the Choose logs to filter out of sinksection, click Add exclusion.
4. Add a filter name.
5. In the Build an exclusion filterbox, add the following:
```
  resource.labels.namespace_name=" TENANT_NAMESPACE 
" 
 
```
6. Click Update sink.

Verify tenant logs

After you have begun using workloads that use TENANT_NAMESPACE , you can verify that the tenant project is receiving tenant-specific logs:

Select the tenant project by using the Google Cloud console project picker.
In the Google Cloud console, go to the Logs Explorer page:

Go to Logs Explorer

If you use the search bar to find this page, then select the result whose subheading is Logging .
In the query-editor field, run the following query:
```
 resource.labels.namespace_name=" TENANT_NAMESPACE 
" 
```
In the Query resultspane, you should see tenant-specific logs that were routed from the main project.

Use tenant logs

In the tenant projects, each team can control how the logs are routed, stored, and analyzed. After the logs are routed to the tenant projects, the individual application teams can choose to route their logs to supported destinations like Logging buckets, or to third-party destinations by using Pub/Sub. For information about routing log entries, see Route logs to supported destinations .

Individual application teams can also set up alerts based on the contents of logs, or metrics derived from logs. For more information, see Monitor your logs .

Clean up

You can remove the objects you created for multi-tenant logging by using gcloud or the Google Cloud console.

gcloud

To remove the objects you created for multi-tenant logging, complete the following steps:

Set variables to simplify the following commands:
```
  export 
  
 TENANT_NAMESPACE 
 = 
 " TENANT_NAMESPACE 
" 
 export 
  
 MAIN_PROJECT 
 = 
 " MAIN_PROJECT_ID 
" 
 export 
  
 TENANT_PROJECT 
 = 
 " TENANT_PROJECT_ID 
" 
 
```
Replace the following:
- TENANT_NAMESPACE : the name for the tenant project namespace
- MAIN-PROJECT-ID : the project ID for your main project
- TENANT-PROJECT-ID : the project ID for your tenant project

Remove the Logs Writer ( roles/logging.logWriter ) role from the service account in the tenant project:

  export 
  
 SERVICE_ACCOUNT 
 = 
 $( 
gcloud  
logging  
sinks  
describe  
gke- $TENANT_NAMESPACE 
-sink  
 \ 
  
--project = 
 $MAIN_PROJECT 
  
 | 
  
 \ 
  
--format = 
 'value(writerIdentity)' 
gcloud  
projects  
remove-iam-policy-binding  
 $TENANT_PROJECT 
  
 \ 
  
--member = 
 $SERVICE_ACCOUNT 
  
 \ 
  
--role = 
 'roles/logging.logWriter' 
  
 \ 
  
--all

Delete the log sink:

 gcloud  
logging  
sinks  
delete  
gke- $TENANT_NAMESPACE 
-sink  
 \ 
  
--project = 
 $MAIN_PROJECT

Delete the namespace:

 kubectl  
delete  
namespace  
 $TENANT_NAMESPACE

Console

In the tenant project, remove the Logs Writer ( roles/logging.logWriter ) role from the service account:
1. In the Google Cloud console, go to the IAM page:
  
  Go to IAM
  
  If you use the search bar to find this page, then select the result whose subheading is IAM & Admin .
2. For the service account you want to delete, click Edit principal.
3. In the Edit accesspanel, click Delete rolenext to the Logs Writer role and click Save.
In the main project, delete the log sink:
1. Use the Google Cloud console project picker to select the tenant GKE project.
2. In the Google Cloud console, go to the Log Router page:
  
  Go to Log Router
  
  If you use the search bar to find this page, then select the result whose subheading is Logging .
3. For the sink you want to delete, click More.
4. Select Delete sink.
5. In the confirmation panel, click Delete.

Limitations

Multi-tenant logging has the following limitations:

The quota for the number of log sinks per project is 200. If you require more than 200 tenants, request a quota increase by opening a support case .
There is a limit of 50 exclusion filters per log bucket. If you intend to have more than 50 tenants, the exclusion filter approach for the _Default bucket needs to be revised. As an alternative, you might do the following:
- Create a single exclusion filter that excludes all non-system or non-default namespaces using this command:
```
 gcloud  
logging  
sinks  
update  
_Default  
 \ 
--project = 
 $MAIN_PROJECT 
  
 \ 
--add-exclusion = 
 "name=gke-all-tenant-default-exclusion,description=\"Exclusion filter on the _Default bucket for all tenants\",filter=resource.labels.namespace_name !~ \"kube\" AND resource.labels.namespace_name !~ \"system\ AND resource.labels.namespace_name != \"Default\"" 
 
```
- Duplicate logs between the tenant project and the main project by not creating the exclusion filter.

What's next

Learn about Best practices for enterprise multi-tenancy on GKE .

Set up multi-tenant logging Stay organized with collections Save and categorize content based on your preferences.

Prerequisites

Configure multi-tenant logging

gcloud

Console

Verify tenant logs

Use tenant logs

Clean up

gcloud

Console

Limitations

What's next

Set up multi-tenant logging