Metadata is crucial for organizing and understanding your data assets. For example, regulated industries need to protect personally identifiable information (PII) like names, addresses, and government issued ID numbers. This data can occur in numerous instances across distributed data systems. Dataplex Universal Catalog can help you to quickly discover and catalog your distributed data assets. You can then classify data as being PII.
By enriching data entries with meaningful context, you can make your data more discoverable and useful. Dataplex Universal Catalog enables context through:
- Aspect type: A JSON template defining related data. For example, for compliance information, you might have PII classification and GDPR compliance.
- Aspect: An instance of an aspect type. For example, {"pii_classification": "confidential", "gdpr_compliant": true}. You can apply an aspect to an entire table or an individual table column.
Once you've classified data, you can apply data quality or access policies to the data.
For more information, see About metadata management in Dataplex Universal Catalog .
Aspects
Aspects let you capture metadata within entries to provide meaningful context. You can use aspects to store:
-
Business metadata: Information that provides business context, such as data classification.
-
Technical metadata: Technical details about the data asset, for example, its schema.
-
Data-derived metadata: Information generated from the data itself, such as statistics from a BigQuery table.
Aspects are considered to be parts of the entry resource and not separate resources. When you modify an aspect, it involves modifying the entry containing the aspect.
You can specify aspects at entry-level for describing an entry, or at column-level for describing a column in an entry.
Every aspect is an instance of an aspect type. An aspect type defines a template for its aspects. Every aspect type contains a set of fields. When you create aspects, you must provide values for those fields.
For a given entry, there can be at most one aspect associated with the entry, per aspect type. You can have multiple aspects associated with entry columns per aspect type.
Categories of aspects
Aspects are categorized into the following:
-
Required aspects: aspects which are mandatory upon creation of an entry. Such aspects are defined by the entry type of a given entry. All entries belonging to an entry type must always have all of the required aspects that are defined by that entry type.
Dataplex Universal Catalog manages the required aspects (for example, schema) for system entries .
Note the following:
-
You can associate required aspects only with entries and not with the columns of an entry.
-
You can't delete the required aspects from an entry.
-
You can read the required aspects of system entries, but can't modify them.
-
Data aspects can't be required aspects.
-
-
Optional aspects: You can associate optional aspects with entries or with entry columns. You can populate optional aspects either at the time of entry creation, or later by updating the entry.
You can delete optional aspects after they have been populated.
Data aspects
Aspects can contain information derived from data, such as results from data profile or example queries. These are referred to as data aspects .
Aspect types
Aspect types are reusable resources that provide templates for aspects.
Categories of aspect types
Aspect types are categorized into custom and system aspect types.
Custom aspect types
Aspect types that you create in Dataplex Universal Catalog are called custom aspect types.
Custom aspect types can be global or regional. You can create custom aspect types
either in a specific regional location (for example, us-central1
) or as a global
resource. The location of an aspect type impacts the scope of its
applicability and determines which entries it can be used with:
- Global aspect types: Can be used to create aspects for entries in any
region. Choose a
globalaspect type if you need to apply the same structure to entries across multiple regions, as you only need to define it once. Because the definition of a global aspect type is replicated across all regions, this option may not be suitable if the aspect type schema itself contains sensitive information or if strict data residency for all metadata components is required. - Regional aspect types: Can only be used to create aspects for entries
that reside in the same region as the aspect type. For example, an aspect
type created in
us-central1can only be used with entries inus-central1. Choose a regional location if an aspect type is only relevant for entries in a specific region, or to ensure that the aspect type definition resides within the selected region, which can help meet data sovereignty and compliance requirements. If you use regional aspect types but need the same structure in multiple regions, you must create and manage separate aspect type definitions in each region.
For more information, see Project and location constraints .
System aspect types
Aspect types that Dataplex Universal Catalog provides, uses, and manages are
called system aspect types. System aspect types are always global, therefore
they can be used in any region. System aspect types are stored in a
Google-managed project with project number 655216118709
. For example, projects/655216118709/locations/global/aspectTypes/schema
.
System aspect types are further categorized into reusable and restricted. The following table describes the categories of system aspect types, and the list of aspect types that Dataplex Universal Catalog provides for each of the categories:
-
contacts -
data-quality-scorecard -
generic -
overview -
schema -
usage
You can read aspects under these aspect types, but can't create or edit aspects under these aspect types.
-
analytics-hub -
aspecttype-aspect -
bigquery-dataset -
bigquery-table -
cloudsql-database -
cloudsql-instance -
cloudsql-table -
entrygroup-aspect -
entrytype-aspect -
sensitive-data-protection-profile -
storage -
storage-bucket -
storage-folder
Before you begin
Before you create and manage aspect types and aspects, complete the tasks described in this section.
Required roles
To get the permissions that you need to create and manage aspect types and aspects, ask your administrator to grant you the following IAM roles on the resource:
- Full set of permissions on metadata resources in Dataplex Universal Catalog, including aspect types and data aspects: Dataplex Catalog Admin
(
roles/dataplex.catalogAdmin) - Create and manage metadata resources in Dataplex Universal Catalog, including aspect types and data aspects: Dataplex Catalog Editor
(
roles/dataplex.catalogEditor) - Full set of permissions on custom aspect types (except for permissions to use aspect types to create or edit entries): Dataplex Aspect Type Owner
(
roles/dataplex.aspectTypeOwner) - View aspect types and IAM policies associated with them: Dataplex Catalog Viewer
(
roles/dataplex.catalogViewer) - Use aspect types to create and modify entries with the corresponding aspects: Dataplex Aspect Type User
(
roles/dataplex.aspectTypeUser) - Add aspects of some of the system aspect types, such as
schema,overview,contacts: Dataplex Entry and EntryLink Owner (roles/dataplex.entryOwner)
For more information about granting roles, see Manage access to projects, folders, and organizations .
You might also be able to get the required permissions through custom roles or other predefined roles .
For more information, see Dataplex Universal Catalog IAM roles .
Enable the API
Enable the Dataplex API in your Dataplex Universal Catalog project.
Create a custom aspect type
Console
-
In the Google Cloud console, go to the Dataplex Universal Catalog Catalogpage.
-
Click the Aspect types & tag templates > Customtab.
-
Click Create aspect type.
-
In the Create aspect typewindow, enter the following:
- Optional: In the Display namefield, enter a name for the aspect type.
- In the Aspect type IDfield, enter a unique ID for the aspect type.
- Optional: In the Descriptionfield, enter a description for the aspect type.
- In the Locationfield, select a location for the aspect type. You can't modify the location of an aspect type after you create it. To understand the impact of choosing a global versus a regional location, see the Custom aspect types section.
-
Optional: Define a template for your aspect type.
In the Templatesection, click Add field. In the New fieldsection, enter the following:
- In the Namefield, enter a name.
- Optional: In the Display namefield, enter a display name.
- Optional: In the Descriptionfield, enter a description.
-
In the Typefield, select a data type for the field. Based on your selection, the next set of fields and options are displayed:
-
If you selected Textas the data type, follow these steps:
- In the Text typefield, select the type of text.
- In the Text valuesfield, provide a hint for the text field. To do this, click Add valueand enter the hint. You can add multiple hints for a text field.
- Click Done.
-
If you selected Enumas the data type, add an enum value:
- Click Add an enum value.
- In the Valuefield, enter an enum value. You can add multiple enum values.
- Click Done.
-
If you selected Arrayas the data type, in the Array itemsection, define the types of items to be present in the array:
- Click Add array item.
- In the Namefield, enter a name for the array items.
- Optional: In the Display namefield, enter a display name for the array items.
- Optional: In the Descriptionfield, enter a description for the array items.
-
In the Typefield, select a data type for the array items.
Based on your selection, the next set of fields and options are displayed. They are similar to the options described for the data types Text, Enum, Map, Array, and Recordelsewhere in this section.
-
Click Done.
-
If you selected Mapas the data type, in the Map valuesection, define the types of values to be present in the map:
- Click Add map value.
- In the Namefield, enter a name for the map.
- Optional: In the Display namefield, enter a display name for the map.
- Optional: In the Descriptionfield, enter a description for the map.
-
In the Typefield, select a data type for the map.
Based on your selection, the next set of fields and options are displayed. They are similar to the options described for the data types Text, Enum, Map, Array, and Recordelsewhere in this section.
-
Click Done.
-
If you selected Recordas the data type, enter the following:
- In the Record IDfield, enter a unique ID that other record fields can use to refer to this record. See the Example for using the Record ID and Record reference fields section of this document.
- Optional: If you want to add a reference to another record from this template, use the Record referencefield. You can't modify this after you create the aspect type. See the Example for using the Record ID and Record reference fields section of this document.
-
In the Record fieldssection, you can define a complex object with multiple nested fields. To do this, click Add record field item, and specify the following:
-
In the Namefield, enter a name for the record field.
-
Optional: In the Display namefield, enter a display name for the record field.
-
Optional: In the Descriptionfield, enter a description for the record field.
-
In the Typefield, select a data type.
Based on your selection, the next set of fields and options are displayed. They are similar to the options described for the data types Text, Enum, Map, Array, and Recordearlier in this section.
-
Click Done.
-
-
To make the field mandatory for an aspect of this type, select Is required. For more information about required aspects and optional aspects, see the categories of aspects section of this document.
-
Click Done.
-
To add multiple fields, click Add fieldand repeat the previous steps.
-
Optional: In the Labelssection, add arbitrary labels as key-value pairs to your resources:
- Click Add label.
- In the Keyfield, enter a key.
- In the Valuefield, enter a value for the key.
- To add more labels, click Add labeland repeat the steps.
-
Click Save.
gcloud
To create a custom aspect type, use the gcloud dataplex aspect-types create
command
.
C#
C#
Before trying this sample, follow the C# setup instructions in the Dataplex Universal Catalog quickstart using client libraries . For more information, see the Dataplex Universal Catalog C# API reference documentation .
To authenticate to Dataplex Universal Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Go
Go
Before trying this sample, follow the Go setup instructions in the Dataplex Universal Catalog quickstart using client libraries . For more information, see the Dataplex Universal Catalog Go API reference documentation .
To authenticate to Dataplex Universal Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Java
Java
Before trying this sample, follow the Java setup instructions in the Dataplex Universal Catalog quickstart using client libraries . For more information, see the Dataplex Universal Catalog Java API reference documentation .
To authenticate to Dataplex Universal Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Python
Python
Before trying this sample, follow the Python setup instructions in the Dataplex Universal Catalog quickstart using client libraries . For more information, see the Dataplex Universal Catalog Python API reference documentation .
To authenticate to Dataplex Universal Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
Ruby
Ruby
Before trying this sample, follow the Ruby setup instructions in the Dataplex Universal Catalog quickstart using client libraries . For more information, see the Dataplex Universal Catalog Ruby API reference documentation .
To authenticate to Dataplex Universal Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
REST
To create a custom aspect type, use the aspectType.create
method.
After you create a custom aspect type, you can add aspects to entries .
Example for using the Record ID and Record reference fields
You can use the Record IDand Record referencefields for recursive references. The following example shows how to use these fields:
Consider an aspect type called Employee
, with the following fields:
- Name
(type:
Text) - Start date
(type:
Date & time) - Designation
(type:
Text) - Current address
(type:
Record) - Permanent address
(type:
Record)
The two address fields Current address
and Permanent address
are of the
same data type Record
. To avoid duplication, you can set the Record IDand Record referencevalues when defining these fields.
When you define the field Current address
, you can specify Record IDas address-field
. For Permanent address
, you can specify the same value
( address-field
) for Record reference. For example:
- Name
(type:
Text) - Start date
(type:
Date & time) - Designation
(type:
Text) - Current address
(type:
Record, Record ID:address-field) - Permanent address
(type:
Record, Record reference:address-field)
This way, you don't need to duplicate the fields of another address.
Add aspects to an entry
After you create an aspect type , you can create aspects of that type. To add aspects to an entry, you must update the entry, as aspects are stored within entries.
Note the following:
- You can add aspects to an entry or to the columns of an entry.
- You can edit the required aspects only for custom entries. You can't delete the required aspects.
- You can edit and delete the optional aspects for both custom entries and system entries.
Console
-
In the Google Cloud console, go to the Dataplex Universal Catalog Searchpage.
-
For Choose search platform, select Dataplex Universal Catalogas the search mode.
-
Search for the entry that you want to add aspects to, and click the entry. The entry details page opens.
-
To add aspects to the entry, follow these steps:
- Click the Detailstab.
- To add required aspects or optional aspects to the entry,
in the Tags & aspectssection, click
Addfor the respective
category.
You can't add required aspects if the entry type of the selected entry has no required aspects defined. - Search for and select the aspect you want to add.
- In the Add aspectwindow, enter the values for the fields.
- Click Save.
-
To add aspects to a column of the entry, follow these steps:
- On the entry details page, click the Schematab.
- Select the columns to which you want to add aspects.
- Click Add aspect.
- Search for and select the aspect you want to add.
- In the Add aspectwindow, enter the values for the fields.
- Click Save.
gcloud
To add aspects to an entry or to a column of an entry, use the gcloud dataplex entries update
command
.
C#
C#
Before trying this sample, follow the C# setup instructions in the Dataplex Universal Catalog quickstart using client libraries . For more information, see the Dataplex Universal Catalog C# API reference documentation .
To authenticate to Dataplex Universal Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
To add aspects to the entry object, see Aspects .
Go
Go
Before trying this sample, follow the Go setup instructions in the Dataplex Universal Catalog quickstart using client libraries . For more information, see the Dataplex Universal Catalog Go API reference documentation .
To authenticate to Dataplex Universal Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .
To add aspects to the entry object, see Entry .
Java
Java
Before trying this sample, follow the Java setup instructions in the Dataplex Universal Catalog quickstart using client libraries . For more information, see the Dataplex Universal Catalog Java API reference documentation .
To authenticate to Dataplex Universal Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment .

