Search for resources in Dataplex Universal Catalog

This document describes how to use search in Dataplex Universal Catalog to search for resources such as BigQuery datasets, Cloud SQL instances, and others.

Dataplex Universal Catalog offers two search modes: keyword search and semantic search (Preview).

Keyword search lets you find resources using specific keywords, filters, and a defined syntax.

Semantic search extends keyword search to support natural language queries. It lets you find resources using everyday language, eliminating the need for complex syntax.

For more information about the Google Cloud assets that are supported in Dataplex Universal Catalog, see Supported Google Cloud sources .

Search scope

The search results in Dataplex Universal Catalog respect permissions that you have over the corresponding resources in source systems.

For example, if you have BigQuery metadata read access to an object, that object appears in your Dataplex Universal Catalog search results. If you have access to a BigQuery table but not to the dataset containing that table, the table still shows up as expected in the Dataplex Universal Catalog search.

By default, search is scoped to your organization. Results include only resources from the same organization as the project you're searching in.

The search results include only those resources that belong to the same VPC-SC perimeter as the project under which search is performed. When using the Google Cloud console, this is the project that is selected in the console.

To broaden the scope of your search results beyond the resources within your project's VPC Service Controls perimeter, use VPC Service Controls ingress and egress rules . These rules facilitate private and efficient data exchange across your organization. You can configure ingress and egress rules using the Google Cloud console or through JSON or YAML files. Refer to the following YAML example and consult the VPC Service Controls documentation to tailor the rule to your specific requirements.

  egressPolicies 
 : 
  
 - 
  
 egressFrom 
 : 
  
 identityType 
 : 
  
 ANY_USER_ACCOUNT 
  
 egressTo 
 : 
  
 # Specify which resources should be present in the search results. In this example, 
  
 # BigQuery. 
  
 operations 
 : 
  
 - 
  
 methodSelectors 
 : 
  
 - 
  
 method 
 : 
  
 '*' 
  
 serviceName 
 : 
  
 bigquery.googleapis.com 
  
 # Specify project ids under which the search is performed. 
  
 resources 
 : 
  
 - 
  
 projects/ SEARCH_PROJECT_ID 
 
 ingressPolicies 
 : 
  
 - 
  
 ingressFrom 
 : 
  
 identityType 
 : 
  
 ANY_USER_ACCOUNT 
  
 sources 
 : 
  
 - 
  
 accessLevel 
 : 
  
 '*' 
  
 ingressTo 
 : 
  
 # Specify which resources should be present in the search results. In this example, 
  
 # BigQuery. 
  
 operations 
 : 
  
 - 
  
 methodSelectors 
 : 
  
 - 
  
 method 
 : 
  
 '*' 
  
 serviceName 
 : 
  
 bigquery.googleapis.com 
  
 # Specify project ids to expose in search results. 
  
 resources 
 : 
  
 - 
  
 projects/ INGRESS_PROJECT_ID

For more information about the Identity and Access Management roles that you need to use Dataplex Universal Catalog search, see Dataplex Universal Catalog IAM roles .

Isolate search results by environment using VPC Service Controls

To isolate Dataplex Universal Catalog search results between environments like development, test, and production, configure separate VPC Service Controls perimeters for each environment. Assign both the projects that contain the data assets and the projects that are used for performing searches to the corresponding environment's perimeter. Searches that are performed from a project within a specific perimeter will only return results for assets that are also located within that same perimeter.

Recall limitations in search

Dataplex Universal Catalog search queries don't guarantee full recall. Results that match your query mightn't be returned. Additionally, returned (and not returned) results can vary if you repeat search queries.

To query all Dataplex Universal Catalog metadata, you can export the metadata to Cloud Storage and then query it from BigQuery. For more information, see Export metadata .

Filters

Filters let you narrow down the search results. Filters are available in both keyword and semantic search, but the available options might differ slightly depending on the mode.

When you provide filters in multiple sections, they are evaluated using the AND logical operator. The search results contain resources that match at least one condition from every selected section. For example, if you select the BigQuery system and the dataset resource type, the search results includes BigQuery datasets but not Vertex AI datasets.

If you select multiple filters within a single section, they are evaluated using the OR logical operator. For example, if you select the dataset resource type and the table resource type, the search results includes both datasets and tables.

For keyword search, filters are grouped into the following sections:

Systemssuch as BigQuery, Cloud SQL, and others. The Dataplex Universal Catalog system contains custom entries.
Aspects (tags)list all aspects available to you.
Projectlists all projects available to you.
Type aliasesdescribe resource types, such as databases, datasets, models, tables, views, services, and custom types.
Datasetscome from BigQuery.

Filter by aspect value

The Aspectsfilters let you query for assets tagged using a specific template. You can use the Customizemenu to further refine results and filter by specific aspect values. The aspect value filter conditions depend on that aspect field's data type. For example, for the datetime and number fields, you can specify a specific date or a range.

Filter visibility

The filters Systems, Type aliases, Project, and Datasetsare displayed depending on the current query in the Searchfield.

For semantic search, the following filters are available:

Scope: search across the organization (default), the current project, or only for starred resources. For more information, see the Search scope section of this document.
Systems: the Google Cloud service that the resource belongs to, such as BigQuery. The Dataplex Universal Catalog system contains entry groups .
Projects: the projects to search in.
Type: the resource type, such as BigQuery connection, Cloud Storage bucket, or database. Depending on the resource type, you can also filter by subtype, such as the connection type or SQL dialect.
Select locations: the locations to search in.
Select datasets: the search results are limited to BigQuery resources that belong to the selected BigQuery datasets. In the Type to filterfield, enter the name of the dataset.
Aspect types: the Dataplex Universal Catalog aspect types that are associated with the resource that you're searching for. To filter by aspect values, click Filter on annotation values, and then select the values.

Before you begin

Before you perform search, ensure that you are granted the required roles and have enabled the necessary API.

Required roles

This section describes the roles and permissions required to search for resources and to access the search results.

For more information about granting roles, see Manage access .

You might also be able to get the required permissions through custom roles or other predefined roles .

Required roles for searching entries

To search for entries, you need at least one of the following IAM roles on the project that is used for search: Dataplex Catalog Admin, Dataplex Catalog Editor, or Dataplex Catalog Viewer. Permissions on search results are checked independently of the selected project.

Required roles for accessing search results

The search results in Dataplex Universal Catalog are scoped according to your role. To search for an asset in Dataplex Universal Catalog, you must have permissions to access the corresponding resource in the source system. For more information, see the Search scope section of this document.

For example, to search for BigQuery datasets, tables, views, and models, you need respective permissions for those entries. For more information, see BigQuery permissions .

The following list describes the minimum permissions required:

To search for a table, you need bigquery.tables.get permission for that table.
To search for a dataset, you need bigquery.datasets.get permission for that dataset.
To search for metadata for a dataset or a table, you need the BigQuery Metadata Viewer role ( roles/bigquery.metadataViewer ).

As another example, to search for Cloud SQL instances, databases, schemas, tables, and views, you need respective permissions on those entries. For more information, see Cloud SQL roles and permissions .

To search for custom entries, you need the Dataplex Catalog Viewer role ( roles/dataplex.catalogViewer ).

Enable the API

Enable the Dataplex API.

Enable the API