Manage data lineage resources using custom constraints

Google Cloud Organization Policy gives you centralized, programmatic control over your organization's resources. As the organization policy administrator , you can define an organization policy, which is a set of restrictions called constraints that apply to Google Cloud resources and descendants of those resources in the Google Cloud resource hierarchy . You can enforce organization policies at the organization, folder, or project level.

Organization Policy provides predefined constraints for various Google Cloud services. However, if you want more granular, customizable control over the specific fields that are restricted in your organization policies, you can also create custom organization policies.

Benefits

You can use a custom organization policy to allow or deny the creation of data lineage processes with conditions based on supported resource attributes, such as process name, source type, and origin.

Policy inheritance

By default, organization policies are inherited by the descendants of the resources on which you enforce the policy. For example, if you enforce a policy on a folder, Google Cloud enforces the policy on all projects in the folder. To learn more about this behavior and how to change it, refer to Hierarchy evaluation rules .

Pricing

The Organization Policy Service, including predefined and custom organization policies, is offered at no charge.

Limitations

• Custom constraints can be enforced only for data lineage Process resources. Other resources, such as Runs and Events , aren't supported.

• Newly enforced custom constraints don't apply to existing resources.

Before you begin

• Ensure that you know your organization ID .

• If you want to test custom organization policies that reference data lineage resources, create a new project. Testing these organization policies in an existing project could disrupt security workflows.

  1. In the Google Cloud console, go to the project selector page.

     Go to project selector

  2. Click Create project .

  3. Name your project. Make a note of your generated project ID.

  4. Edit the other fields as needed.

  5. Click Create .

Required roles

To get the permissions that you need to manage organization policies, ask your administrator to grant you the Organization Policy Administrator ( roles/orgpolicy.policyAdmin ) IAM role on the organization. For more information about granting roles, see Manage access to projects, folders, and organizations .

This predefined role contains the permissions required to manage organization policies. To see the exact permissions that are required, expand the Required permissionssection:

You might also be able to get these permissions with custom roles or other predefined roles .

Create a custom constraint

A custom constraint is defined in a YAML file by the resources, methods, conditions, and actions that are supported by the service on which you are enforcing the organization policy. Conditions for your custom constraints are defined using Common Expression Language (CEL) . For more information about how to build conditions in custom constraints using CEL, see the CEL section of Creating and managing custom constraints .

To create a YAML file for a custom constraint:

  name 
 : 
  
 organizations/ ORGANIZATION_ID 
/customConstraints/ CONSTRAINT_NAME 
 
 resourceTypes 
 : 
 - 
  
 datalineage.googleapis.com/ RESOURCE_TYPE 
 
 methodTypes 
 : 
 - 
  
 CREATE 
 - 
  
 UPDATE 
 condition 
 : 
  
 " CONDITION 
" 
 actionType 
 : 
  
  ACTION 
 
 displayName 
 : 
  
  DISPLAY_NAME 
 
 description 
 : 
  
  DESCRIPTION 
 
 

Replace the following:

• ORGANIZATION_ID : your organization ID, such as 123456789 .

• CONSTRAINT_NAME : the name you want for your new custom constraint. A custom constraint must start with custom. , and can only include uppercase letters, lowercase letters, or numbers—for example, custom.denyLineageProcess . The maximum length of this field is 70 characters, not counting the prefix—for example, organizations/123456789/customConstraints/custom .

• RESOURCE_TYPE : the name (not the URI) of the Data Lineage API REST resource containing the object and field you want to restrict. For data lineage only Process is available.

• CONDITION : a CEL condition that is written against a representation of a supported service resource. This field has a maximum length of 1000 characters. See Supported resources for more information about the resources available to write conditions against. For example, "resource.name.contains('invalid_name')" .

• ACTION : the action to take if the condition is met. This can be either ALLOW or DENY .

• DISPLAY_NAME : a human-friendly name for the constraint. This field has a maximum length of 200 characters.

• DESCRIPTION : a human-friendly description of the constraint to display as an error message when the policy is violated. This field has a maximum length of 2000 characters.

For more information about how to create a custom constraint, see Defining custom constraints .

Set up a custom constraint

gcloud  
org-policies  
set-custom-constraint  
 CONSTRAINT_PATH 

gcloud  
org-policies  
list-custom-constraints  
--organization = 
 ORGANIZATION_ID 

Enforce a custom organization policy

  
 name 
 : 
  
 projects/ PROJECT_ID 
/policies/ CONSTRAINT_NAME 
 
  
 spec 
 : 
  
 rules 
 : 
  
 - 
  
 enforce 
 : 
  
 true 
  

  
gcloud  
org-policies  
set-policy  
 POLICY_PATH 
  

Test the custom organization policy

This section describes how to test an organization policy constraint that restricts users from creating a data lineage process with a name that contains the text invalid_name .

If you want to test this custom constraint, do the following:

1. Create a new project, as described in the Before you begin section.

2. Copy the following constraint into a YAML file:

      
    name 
    : 
     
    organizations/ ORGANIZATION_ID 
   /customConstraints/custom.denyLineageProcess 
     
    resourceTypes 
    : 
     
    datalineage.googleapis.com/Process 
     
    methodTypes 
    : 
     
    - 
     
    CREATE 
     
    - 
     
    UPDATE 
     
    condition 
    : 
     
    "resource.name.contains('invalid_name')" 
     
    actionType 
    : 
     
    DENY 
     
    displayName 
    : 
     
    Do not allow data lineage process with 'invalid_name' to be created. 
    
   

   Replace ORGANIZATION_ID with the numeric ID of your Google Cloud organization.

3. Set up the custom constraint and enforce it for the project that you created to test the custom organization policy constraint.

4. Create a process with the name that you included in the custom constraint.

   curl  
   -s  
   -X  
   POST  
   -H  
    "Authorization: Bearer 
    $( 
   gcloud  
   auth  
   print-access-token ) 
    " 
     
   -H  
    "Content-Type: application.json" 
     
    \ 
    "https://datalineage.googleapis.com/v1/projects/ PROJECT_ID 
   /locations/ LOCATION 
   /processes" 
     
    \ 
   -d  
    '{"name":"projects/ PROJECT_ID 
   /locations/ LOCATION 
   /processes/invalid_name"}' 
   

   Replace the following:

   • PROJECT_ID : ID of the project where the process is created. It must be in the same organization that the constraint is defined in.
   • LOCATION : Google Cloud region where the process is created.

   The output is the following:

    Operation denied by org policy on resource 'projects/ PROJECT_ID 
   /locations/ LOCATION 
   ': ["customConstraints/custom.denyLineageProcess"] 
   

Data lineage supported resources and operations

The following custom constraint fields are available to use when you create or update a data lineage process:

• resource.name
• resource.displayName
• resource.origin.name
• resource.origin.sourceType

Custom constraints are checked for the following methods:

• CreateProcess
• UpdateProcess
• ProcessopenLineageRunEvent

Example custom organization policies for common use cases

The following table provides the syntax of some custom constraints for common use cases:

For more information about CEL macros available for use in custom constraint conditions, see Common Expression Language .

  
 name 
 : 
  
 organizations/ ORGANIZATION_ID 
/customConstraints/custom.denyLineageProcessesCreation 
  
 resourceTypes 
 : 
  
 - 
  
 datalineage.googleapis.com/Process 
  
 methodTypes 
 : 
  
 - 
  
 CREATE 
  
 condition 
 : 
  
 "True" 
  
 actionType 
 : 
  
 DENY 
  
 displayName 
 : 
  
 Deny creation of all data lineage processes. 

  
 name 
 : 
  
 organizations/ ORGANIZATION_ID 
/customConstraints/custom.denyLineageProcessesCreation 
  
 resourceTypes 
 : 
  
 - 
  
 datalineage.googleapis.com/Process 
  
 methodTypes 
 : 
  
 - 
  
 CREATE 
  
 condition 
 : 
  
 "resource.origin.sourceType 
  
 == 
  
 'BIGQUERY' 
  
 || 
  
 resource.origin.sourceType 
  
 == 
  
 'DATAPROC'" 
  
 actionType 
 : 
  
 DENY 
  
 displayName 
 : 
  
 Deny data lineage processes created by BigQuery or Dataproc. 

  
 name 
 : 
  
 organizations/ ORGANIZATION_ID 
/customConstraints/custom.denyLineageProcessesCreation 
  
 resourceTypes 
 : 
  
 - 
  
 datalineage.googleapis.com/Process 
  
 methodTypes 
 : 
  
 - 
  
 CREATE 
  
 - 
  
 UPDATE 
  
 condition 
 : 
  
 "resource.name.contains(' RESTRICTED_NAME 
')" 
  
 actionType 
 : 
  
 DENY 
  
 displayName 
 : 
  
 Deny data lineage processes whose name contains RESTRICTED_NAME 
. 

What's next

• See Introduction to the Organization Policy Service to learn more about organization policies.
• Learn more about how to create and manage organization policies .
• See the full list of predefined organization policy constraints .