View data lineage for Google Cloud systems

View data lineage to understand the relationships between your project's resources and the processes that created them. These relationships show how data assets, such as tables and datasets, are transformed by processes like queries and pipelines. This guide describes how to view data lineage details in the Google Cloud console or retrieve them by using the Data Lineage API.

Roles and permissions

Data lineage tracks lineage information automatically when you enable the Data Lineage API . You don't need any administrator or editor roles to capture lineage for your data assets.

To view data lineage, you need specific Identity and Access Management (IAM) permissions. Lineage information is captured across projects, so you need permissions in multiple projects.

• When viewing lineage in Knowledge Catalog, BigQuery, or Vertex AI: you need permissions to view lineage information in the project where you are viewing it.

• When viewing lineage that was recorded in other projects: you need permissions to view lineage information in those projects where it was recorded.

To get the permissions that you need to view data lineage, ask your administrator to grant you the following IAM roles:

• Data Lineage Viewer ( roles/datalineage.viewer ) on the project where lineage is recorded, and the project where lineage is viewed
• View BigQuery table details: BigQuery Data Viewer ( roles/bigquery.dataViewer ) on the table's storage project
• View BigQuery job details: BigQuery Resource Viewer ( roles/bigquery.resourceViewer ) on the job's compute project
• View details for other cataloged assets: Dataplex Catalog Viewer ( roles/dataplex.catalogViewer ) on the project where catalog entries are stored

For more information about granting roles, see Manage access to projects, folders, and organizations .

These predefined roles contain the permissions required to view data lineage. To see the exact permissions that are required, expand the Required permissionssection:

You might also be able to get these permissions with custom roles or other predefined roles .

Types of data lineage views

You can view lineage information as a graph or a list. By default, the lineage graph displays table-level lineage. For BigQuery and Managed Service for Apache Spark jobs, you can view column-level lineage in both graph and list views.

The following view types are available:

• Graph view: displays lineage as an interactive graph, letting you explore relationships between data assets and columns by expanding nodes.

• List view: displays lineage in a tabular format, providing simplified and detailed representations of table-level and column-level lineage. You can customize columns and export lineage data from this view.

The key elements in the graph are described as follows:

• Nodes: represent the data entities. In the table-level view, a node shows the table name and its columns. In the column-level view, each node represents a specific table and its columns that have lineage.

• Edges: the lines that connect nodes and represent the processes that occur between them. Edges can feature icons or labels to provide more information about the transformation:

  • Icons: In table-level view, icons appear on edges to represent the transformation process. When you manually explore the graph, icons on edges represent the source system of the process (for example, BigQuery or Vertex AI). If multiple processes are involved, a 'multiple processes' icon is displayed. If the process source system is unknown, a gear icon is used. When you apply filters, a gear icon is used for all processes.
  • Labels: In column-level view, edges are labeled to describe the type of dependency between columns, such as Exact copy or Other .

Enable data lineage

Enable data lineage to begin automatically tracking lineage information for supported systems . By default, enabling the API activates lineage tracking for most supported services. To control Managed Service for Apache Spark lineage ingestion, see Control lineage ingestion for a service .

You must enable the Data Lineage API in both the project where you view lineage and the projects where lineage is recorded. For more information, see Project types .

1. To capture lineage information, complete the following steps:

   1. In the Google Cloud console, on the Project selectorpage, select the project where you want to record lineage.

      Go to Project selector

   2. Enable the Data Lineage API.

      Enable the Data Lineage API

   3. Repeat the previous steps for each project where you want to record lineage.

2. In the project where you view lineage, enable the Data Lineage API and the Dataplex API.

   Enable the APIs

Control lineage ingestion for a service

After you enable the Data Lineage API, the service starts automatic lineage tracking for most supported services. You can then selectively enable or disable lineage ingestion for specific integrations at the project, folder, or organization level. During preview, this feature supports configuring ingestion for Managed Service for Apache Spark, BigQuery, and Managed Service for Apache Airflow.

The configuration is hierarchical. The most specific configuration takes precedence. For example, a project-level configuration overrides a folder-level configuration. If no configuration is set, the service's default behavior is used. For Managed Service for Apache Spark, BigQuery, and Managed Airflow, the default is Enabled.

Any changes to the configuration might take up to 24 hours to propagate, but usually become effective within two hours.

For more information about controlling lineage ingestion including how the configuration is applied hierarchically, see Control lineage ingestion .

Prerequisites

To control lineage ingestion, you must use the Data Lineage API. Ensure you have a client project configured for billing and quota, as the Data Lineage API is a client-based API .

1. Enable the datalineage.googleapis.com API in your client project. For more information, see Enable data lineage .

2. Set the client project. For the following examples, use the X-Goog-User-Project header. For more information, see System parameters .

Get current configuration

To check whether lineage ingestion is enabled for a resource or to obtain the etag value before you modify the configuration, retrieve the current configuration.

  using 
  
  Google.Cloud.DataCatalog.Lineage.ConfigManagement.V1 
 
 ; 
 public 
  
 sealed 
  
 partial 
  
 class 
  
 GeneratedConfigManagementServiceClientSnippets 
 { 
  
 /// <summary>Snippet for GetConfig</summary> 
  
 /// <remarks> 
  
 /// This snippet has been automatically generated and should be regarded as a code template only. 
  
 /// It will require modifications to work: 
  
 /// - It may require correct/in-range values for request initialization. 
  
 /// - It may require specifying regional endpoints when creating the service client as shown in 
  
 ///   https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint. 
  
 /// </remarks> 
  
 public 
  
 void 
  
 GetConfigRequestObject 
 () 
  
 { 
  
 // Create client 
  
  ConfigManagementServiceClient 
 
  
 configManagementServiceClient 
  
 = 
  
  ConfigManagementServiceClient 
 
 . 
  Create 
 
 (); 
  
 // Initialize request argument(s) 
  
  GetConfigRequest 
 
  
 request 
  
 = 
  
 new 
  
  GetConfigRequest 
 
  
 { 
  
 ConfigName 
  
 = 
  
  ConfigName 
 
 . 
  FromProjectLocation 
 
 ( 
 "[PROJECT]" 
 , 
  
 "[LOCATION]" 
 ), 
  
 }; 
  
 // Make the request 
  
  Config 
 
  
 response 
  
 = 
  
 configManagementServiceClient 
 . 
  GetConfig 
 
 ( 
 request 
 ); 
  
 } 
 }