This document describes how to integrate data sources with Dataplex Universal Catalog. Dataplex Universal Catalog can import and maintain up-to-date metadata from Google Cloud data sources.
With metadata ingested, Dataplex Universal Catalog does the following:
- Makes the existing metadata discoverable through search. For more information, see Search for data assets in Dataplex Universal Catalog .
- Allows the members of your organization to enrich your data with additional business metadata by using aspects. For more information, see Manage aspects and enrich metadata .
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project : Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
- Create a project
: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles .
-
Verify that billing is enabled for your Google Cloud project .
-
Enable the Dataplex API.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles . -
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project : Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
- Create a project
: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles .
-
Verify that billing is enabled for your Google Cloud project .
-
Enable the Dataplex API.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles .
Integrate AlloyDB for PostgreSQL
After you enable the Dataplex Universal Catalog integration on your AlloyDB for PostgreSQL cluster Preview , Dataplex Universal Catalog automatically retrieves metadata from AlloyDB for PostgreSQL resources, including clusters, instances, databases, tables, columns, and views.
Dataplex Universal Catalog retrieves metadata only from AlloyDB for PostgreSQL primary instances and not from read replicas.
For more information, see Manage your AlloyDB for PostgreSQL resources using Dataplex Universal Catalog .
Integrate Cloud SQL
After you enable the Dataplex Universal Catalog integration on your Cloud SQL instance , Dataplex Universal Catalog automatically retrieves metadata from Cloud SQL resources, including instances, databases, tables, columns, and views.
Dataplex Universal Catalog retrieves metadata only from Cloud SQL primary instances and not from read replicas.
For more information, see Manage your Cloud SQL resources using Dataplex Universal Catalog .
What's next
- Learn how to search for resources in Dataplex Universal Catalog.
- Learn more about metadata management in Dataplex Universal Catalog .
- Learn how to enrich entries with metadata by using aspects .
- Learn how to manage entries and ingest custom sources .
