Create and use data profile scans

Dataplex Universal Catalog lets you identify common statistical characteristics (common values, data distribution, null counts) of the columns in your BigQuery tables. This information helps you to understand and analyze your data more effectively.

For more information about Dataplex Universal Catalog data profile scans, see About data profiling .

Before you begin

Enable the Dataplex API.

Enable the API

Required roles and permissions

This section describes the IAM roles and permissions needed to use Dataplex Universal Catalog data profile scans.

User roles and permissions

To get the permissions that you need to create and manage data profile scans, ask your administrator to grant you the following IAM roles :

• Create, run, update, and delete data profile scans: Dataplex DataScan Editor ( roles/dataplex.dataScanEditor ) on the project containing the data scan
• View data profile scan results, jobs, and history: Dataplex DataScan Viewer ( roles/dataplex.dataScanViewer ) on the project containing the data scan
• Publish data profile scan results to Dataplex Universal Catalog: Dataplex Catalog Editor ( roles/dataplex.catalogEditor ) on the @bigquery entry group
• View published data profile scan results in BigQuery on the Data profiletab: BigQuery Data Viewer ( roles/bigquery.dataViewer ) on the table

For more information about granting roles, see Manage access to projects, folders, and organizations .

These predefined roles contain the permissions required to create and manage data profile scans. To see the exact permissions that are required, expand the Required permissionssection:

You might also be able to get these permissions with custom roles or other predefined roles .

Dataplex Universal Catalog service account roles and permissions

To ensure that the Dataplex Universal Catalog service account has the necessary permissions to run data profile scans and export results, ask your administrator to grant the following IAM roles to the Dataplex Universal Catalog service account:

• Run data profile scans against BigQuery data:
  • BigQuery Job User ( roles/bigquery.jobUser ) on project running the scan
  • BigQuery Data Viewer ( roles/bigquery.dataViewer ) on tables being scanned
• Run data profile scans for BigQuery external tables that use Cloud Storage data:
  • Storage Object Viewer ( roles/storage.objectViewer ) on Cloud Storage bucket
  • Storage Legacy Bucket Reader ( roles/storage.legacyBucketReader ) on Cloud Storage bucket
• Export data profile scan results to a BigQuery table: BigQuery Data Editor ( roles/bigquery.dataEditor ) on table

For more information about granting roles, see Manage access to projects, folders, and organizations .

These predefined roles contain the permissions required to run data profile scans and export results. To see the exact permissions that are required, expand the Required permissionssection:

Your administrator might also be able to give the Dataplex Universal Catalog service account these permissions with custom roles or other predefined roles .

If a table uses BigQuery row-level security , then Dataplex Universal Catalog can only scan rows visible to the Dataplex Universal Catalog service account. To allow Dataplex Universal Catalog to scan all rows, add its service account to a row filter where the predicate is TRUE .

If a table uses BigQuery column-level security , then Dataplex Universal Catalog requires access to scan protected columns. To grant access, give the Dataplex Universal Catalog service account the Data Catalog Fine-Grained Reader( roles/datacatalog.fineGrainedReader ) role on all policy tags used in the table. The user creating or updating a data scan also needs permissions on protected columns.

Grant roles to the Dataplex Universal Catalog service account

To run data profile scans, Dataplex Universal Catalog uses a service account that requires permissions to run BigQuery jobs and read BigQuery table data. To grant the required roles, follow these steps:

1. Get the Dataplex Universal Catalog service account email address. If you haven't created a data profile or data quality scan in this project before, run the following gcloud command to generate the service identity:

    gcloud  
   beta  
   services  
   identity  
   create  
   --service = 
   dataplex.googleapis.com 
   

   The command returns the service account email, which has the following format: service- PROJECT_ID @gcp-sa-dataplex.iam.gserviceaccount.com.

   If the service account already exists, you can find its email by viewing principals with the Dataplexname on the IAMpage in the Google Cloud console.

2. Grant the service account the BigQuery Job User( roles/bigquery.jobUser ) role on your project. This role lets the service account run BigQuery jobs for the scan.

    gcloud  
   projects  
   add-iam-policy-binding  
    PROJECT_ID 
     
    \ 
     
   --member = 
    "serviceAccount:service- PROJECT_NUMBER 
   @gcp-sa-dataplex.iam.gserviceaccount.com" 
     
    \ 
     
   --role = 
    "roles/bigquery.jobUser" 
    
   

   Replace the following:

   • PROJECT_ID : your Google Cloud project ID.
   • service- PROJECT_NUMBER @gcp-sa-dataplex.iam.gserviceaccount.com : the email of the Dataplex Universal Catalog service account.

3. Grant the service account the BigQuery Data Viewer( roles/bigquery.dataViewer ) role for each table that you want to profile. This role grants read-only access to the tables.

    gcloud  
   bigquery  
   tables  
   add-iam-policy-binding  
    DATASET_ID 
   . TABLE_ID 
     
    \ 
     
   --member = 
    "serviceAccount:service- PROJECT_NUMBER 
   @gcp-sa-dataplex.iam.gserviceaccount.com" 
     
    \ 
     
   --role = 
    "roles/bigquery.dataViewer" 
    
   

   Replace the following:

   • DATASET_ID : the ID of the dataset containing the table.
   • TABLE_ID : the ID of the table to profile.
   • service- PROJECT_NUMBER @gcp-sa-dataplex.iam.gserviceaccount.com : the email of the Dataplex Universal Catalog service account.

Create a data profile scan

gcloud dataplex datascans create data-profile DATASCAN 
\
--location= LOCATION 
\
--data-source-entity= DATA_SOURCE_ENTITY 

gcloud dataplex datascans create data-profile DATASCAN 
\
--location= LOCATION 
\
--data-source-resource= DATA_SOURCE_RESOURCE 

  using 
  
  Google.Api.Gax.ResourceNames 
 
 ; 
 using 
  
  Google.Cloud.Dataplex.V1 
 
 ; 
 using 
  
  Google.LongRunning 
 
 ; 
 public 
  
 sealed 
  
 partial 
  
 class 
  
 GeneratedDataScanServiceClientSnippets 
 { 
  
 /// <summary>Snippet for CreateDataScan</summary> 
  
 /// <remarks> 
  
 /// This snippet has been automatically generated and should be regarded as a code template only. 
  
 /// It will require modifications to work: 
  
 /// - It may require correct/in-range values for request initialization. 
  
 /// - It may require specifying regional endpoints when creating the service client as shown in 
  
 ///   https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint. 
  
 /// </remarks> 
  
 public 
  
 void 
  
 CreateDataScanRequestObject 
 () 
  
 { 
  
 // Create client 
  
  DataScanServiceClient 
 
  
 dataScanServiceClient 
  
 = 
  
  DataScanServiceClient 
 
 . 
  Create 
 
 (); 
  
 // Initialize request argument(s) 
  
  CreateDataScanRequest 
 
  
 request 
  
 = 
  
 new 
  
  CreateDataScanRequest 
 
  
 { 
  
 ParentAsLocationName 
  
 = 
  
  LocationName 
 
 . 
  FromProjectLocation 
 
 ( 
 "[PROJECT]" 
 , 
  
 "[LOCATION]" 
 ), 
  
 DataScan 
  
 = 
  
 new 
  
  DataScan 
 
 (), 
  
 DataScanId 
  
 = 
  
 "" 
 , 
  
 ValidateOnly 
  
 = 
  
 false 
 , 
  
 }; 
  
 // Make the request 
  
 Operation<DataScan 
 , 
  
 OperationMetadata 
>  
 response 
  
 = 
  
 dataScanServiceClient 
 . 
  CreateDataScan 
 
 ( 
 request 
 ); 
  
 // Poll until the returned long-running operation is complete 
  
 Operation<DataScan 
 , 
  
 OperationMetadata 
>  
 completedResponse 
  
 = 
  
 response 
 . 
 PollUntilCompleted 
 (); 
  
 // Retrieve the operation result 
  
  DataScan 
 
  
 result 
  
 = 
  
 completedResponse 
 . 
 Result 
 ; 
  
 // Or get the name of the operation 
  
 string 
  
 operationName 
  
 = 
  
 response 
 . 
 Name 
 ; 
  
 // This name can be stored, then the long-running operation retrieved later by name 
  
 Operation<DataScan 
 , 
  
 OperationMetadata 
>  
 retrievedResponse 
  
 = 
  
 dataScanServiceClient 
 . 
  PollOnceCreateDataScan 
 
 ( 
 operationName 
 ); 
  
 // Check if the retrieved long-running operation has completed 
  
 if 
  
 ( 
 retrievedResponse 
 . 
 IsCompleted 
 ) 
  
 { 
  
 // If it has completed, then access the result 
  
  DataScan 
 
  
 retrievedResult 
  
 = 
  
 retrievedResponse 
 . 
 Result 
 ; 
  
 } 
  
 } 
 }