Manage metadata of Dataform repositories with Knowledge Catalog

This document describes using Knowledge Catalog to view metadata of Dataform repositories and annotate them with aspects.

Knowledge Catalog lets you store and manage your metadata. Dataform repositories are available in Knowledge Catalog by default, without any additional configuration. Knowledge Catalog automatically retrieves the following metadata from Dataform repositories:

Data asset name
Data asset parent
Data asset location
Data asset type
Corresponding Google Cloud project
Third-party repository source
Service agent or service account

Before you begin

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project : Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
- Create a project : To create a project, you need the Project Creator role ( roles/resourcemanager.projectCreator ), which contains the resourcemanager.projects.create permission. Learn how to grant roles .
Note : If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector
Enable the Dataplex API.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role ( roles/serviceusage.serviceUsageAdmin ), which contains the serviceusage.services.enable permission. Learn how to grant roles .

Enable the API
Ensure that you have the necessary Identity and Access Management (IAM) permissions .

Required roles

To manage Dataform repositories in Knowledge Catalog, you need the predefined Dataform roles that give you access to Dataform resources, and required Knowledge Catalog roles .

Manage metadata in Knowledge Catalog

Knowledge Catalog logs Dataform repositories as entries with the following entry values:

System entry group: The system entry group for Dataform repositories is @dataform . To view details of repository entries in Knowledge Catalog, you need to view the dataform system entry group. For instructions about how to view a list of all entries in an entry group, see View details of an entry group in the Knowledge Catalog documentation.
System entry type: The system entry type for Dataform repositories is dataform-repository . To view details of Dataform repositories, you need to view the dataform-repository system entry type. For instructions about how to view details of a selected entry type, see View details of an entry type in the Knowledge Catalog documentation. For instructions about how to view details of a selected entry, see View details of an entry in the Knowledge Catalog documentation.

Search for assets

For instructions about how to search for data assets in Knowledge Catalog, see Search for data assets in Knowledge Catalog in the Knowledge Catalog documentation.

Annotate repositories

To provide additional context to Dataform repositories in Knowledge Catalog, you can annotate Dataform entries with aspects .

The system aspect type for Dataform repositories is dataform-repository .

For instructions about how to annotate entries with aspects, see Manage aspects and enrich metadata in the Knowledge Catalog documentation.

Limitations

Managing Dataform repositories in Knowledge Catalog is available with the following limitations:

You cannot view or manage Dataform file-level assets in Knowledge Catalog—for example, tables, or views.

Locations

You can use Knowledge Catalog to manage Dataform repositories from all Dataform locations .

Quotas

Managing Dataform repositories in Knowledge Catalog is subject to Knowledge Catalog quotas and limits .

Pricing

Managing Dataform repositories in Knowledge Catalog is subject to Knowledge Catalog pricing .

What's next

To learn more about Knowledge Catalog, see Knowledge Catalog overview in the Knowledge Catalog documentation.