Manage entries and ingest custom sources

This document explains how to bring information about your unique, custom data sources into Knowledge Catalog (formerly Dataplex Universal Catalog).

Knowledge Catalog serves as a central registry for all your data assets. To integrate a custom source, you need three key components:

• Entry group: This is a secured container that holds your custom data records. Think of it as a folder that defines access permissions and the geographic location (region) for your records.

• Entry type: This acts as a template or schema. It defines the required metadata fields (like data owner, update frequency, or quality score) for a specific kind of data asset.

• Entry: This is the individual record or asset itself (for example, a specific proprietary file or database table).

The integration process

To register a custom data source, you need to set up this structure:

1. Create an entry group (the folder).

2. Define an entry type (the template/schema).

3. Create the custom entry (the data record) using the template within the folder.

Entries

An entry represents a data asset that you capture metadata for. Every entry is an instance of an entry type. Each operation on aspects for an entry needs to comply with the required aspects of its entry type. For example, when you create an entry, you must provide values for all the aspect types defined by the entry type. You can't delete those aspects for an entry that are marked as required in the entry type.

Categories of entries

• System entries: Knowledge Catalog creates entries for Google Cloud resources, such as BigQuery datasets or tables. The entries that Knowledge Catalog creates are called system entries. Knowledge Catalog automatically keeps system entries up-to-date through continuous metadata synchronization from the supported Google Cloud systems.

  You can't modify the metadata that is populated automatically (called required aspects) for system entries. You can only add and modify additional metadata for system entries by using optional aspects. For more information, see Categories of aspects .

• Custom entries: entries that you can create and manage for custom resources are called custom entries. Custom resources are the resources in non-Google services that are hosted either in Google Cloud or externally (for example, on-premises).

Entry groups

An entry group is a container for one or more entries. You can use entry groups to manage access control and regional location for the entries. Every entry group belongs to a project.

Categories of entry groups

• System entry groups: for Google Cloud resources, Knowledge Catalog automatically creates entry groups for each system in every project and location where the resources are. For example, @bigquery is the system entry group for BigQuery.

• Custom entry groups: Entry groups that you create for custom resources.

Entry types

Entry types define the required metadata for entries of this type, using a set of required aspect types.

You can specify the required aspect types only on entries, and not on the columns of an entry. When you create an entry of a specific entry type, you must provide values for all required aspect types that are specified by the entry type.

Required aspect types that are referenced within an entry type must belong to the same project as the entry type.

Categories of entry types

• Custom entry types: Entry types that you create and manage. You can use these entry types to create custom entries.

• System entry types: Knowledge Catalog provides these entry types by default. System entry types are further categorized into reusable and restricted.

  The following table describes the categories of system entry types, and the list of entry types that Knowledge Catalog provides for each of the categories:

  Category of system entry type

  Description

  Entry types that Knowledge Catalog provides

  Reusable system entry type

  You can use this entry type to create custom entries.

  • generic

  Restricted system entry type

  These are reserved for system use, such as creating entries for Google Cloud resources.
  
  You can't use these entry types to create entries, but you can edit entries of these entry types to add optional aspects.

  Including but not limited to:

  • analytics-hub-exchange
  • analytics-hub-listing
  • aspecttype
  • bigquery-dataset
  • bigquery-table
  • cloudsql-database
  • cloudsql-instance
  • cloudsql-table
  • entrygroup
  • entrytype
  • sql-access
  • storage
  • storage-bucket
  • storage-folder

You can create a custom entry type in a specific regional location or as a global resource. The location of an entry type impacts the scope of its applicability. For more information, see Project and location constraints .

System entry types are always global. System entry types are stored in a Google-managed project with project ID 655216118709 . For example, projects/655216118709/locations/global/entryTypes/bigquery-table .

Before you begin

Before you manage entries and ingest custom data sources, ensure that you have completed the tasks described in this section.

Required roles

To get the permissions that you need to create and manage entries, ask your administrator to grant you the following IAM roles on the resource:

• Full set of permissions on entries, entry groups, and entry types: Dataplex Catalog Admin ( roles/dataplex.catalogAdmin )
• Create and manage metadata resources in Knowledge Catalog, including entries, entry groups, and entry types: Dataplex Catalog Editor ( roles/dataplex.catalogEditor )
• Full access to entry-related operations: Dataplex Entry and EntryLink Owner ( roles/dataplex.entryOwner )
• View entries, entry groups, and entry types, and IAM policies associated with them: Dataplex Catalog Viewer ( roles/dataplex.catalogViewer )
• Use aspect types to create aspects of those types, when creating custom entries: Dataplex Aspect Type User ( roles/dataplex.aspectTypeUser )
• Create and manage entry groups: Dataplex Entry Group Owner ( roles/dataplex.entryGroupOwner )
• Create and manage entry types: Dataplex Entry Type Owner ( roles/dataplex.entryTypeOwner )
• Use entry types to create and modify entries of those types: Dataplex Entry Type User ( roles/dataplex.entryTypeUser )

For more information about granting roles, see Manage access to projects, folders, and organizations .

You might also be able to get the required permissions through custom roles or other predefined roles .

For more information, see Knowledge Catalog IAM roles .

Enable the API

Enable the Dataplex API in your Knowledge Catalog project. If you are using a new Google Cloud project or a project in which the BigQuery API is enabled, the Dataplex API is enabled by default.

Enable the API

Create an entry group

  using 
  
  Google.Api.Gax.ResourceNames 
 
 ; 
 using 
  
  Google.Cloud.Dataplex.V1 
 
 ; 
 using 
  
  Google.LongRunning 
 
 ; 
 public 
  
 sealed 
  
 partial 
  
 class 
  
 GeneratedCatalogServiceClientSnippets 
 { 
  
 /// <summary>Snippet for CreateEntryGroup</summary> 
  
 /// <remarks> 
  
 /// This snippet has been automatically generated and should be regarded as a code template only. 
  
 /// It will require modifications to work: 
  
 /// - It may require correct/in-range values for request initialization. 
  
 /// - It may require specifying regional endpoints when creating the service client as shown in 
  
 ///   https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint. 
  
 /// </remarks> 
  
 public 
  
 void 
  
 CreateEntryGroupRequestObject 
 () 
  
 { 
  
 // Create client 
  
  CatalogServiceClient 
 
  
 catalogServiceClient 
  
 = 
  
  CatalogServiceClient 
 
 . 
  Create 
 
 (); 
  
 // Initialize request argument(s) 
  
  CreateEntryGroupRequest 
 
  
 request 
  
 = 
  
 new 
  
  CreateEntryGroupRequest 
 
  
 { 
  
 ParentAsLocationName 
  
 = 
  
  LocationName 
 
 . 
  FromProjectLocation 
 
 ( 
 "[PROJECT]" 
 , 
  
 "[LOCATION]" 
 ), 
  
 EntryGroupId 
  
 = 
  
 "" 
 , 
  
 EntryGroup 
  
 = 
  
 new 
  
  EntryGroup 
 
 (), 
  
 ValidateOnly 
  
 = 
  
 false 
 , 
  
 }; 
  
 // Make the request 
  
 Operation<EntryGroup 
 , 
  
 OperationMetadata 
>  
 response 
  
 = 
  
 catalogServiceClient 
 . 
  CreateEntryGroup 
 
 ( 
 request 
 ); 
  
 // Poll until the returned long-running operation is complete 
  
 Operation<EntryGroup 
 , 
  
 OperationMetadata 
>  
 completedResponse 
  
 = 
  
 response 
 . 
 PollUntilCompleted 
 (); 
  
 // Retrieve the operation result 
  
  EntryGroup 
 
  
 result 
  
 = 
  
 completedResponse 
 . 
 Result 
 ; 
  
 // Or get the name of the operation 
  
 string 
  
 operationName 
  
 = 
  
 response 
 . 
 Name 
 ; 
  
 // This name can be stored, then the long-running operation retrieved later by name 
  
 Operation<EntryGroup 
 , 
  
 OperationMetadata 
>  
 retrievedResponse 
  
 = 
  
 catalogServiceClient 
 . 
  PollOnceCreateEntryGroup 
 
 ( 
 operationName 
 ); 
  
 // Check if the retrieved long-running operation has completed 
  
 if 
  
 ( 
 retrievedResponse 
 . 
 IsCompleted 
 ) 
  
 { 
  
 // If it has completed, then access the result 
  
  EntryGroup 
 
  
 retrievedResult 
  
 = 
  
 retrievedResponse 
 . 
 Result 
 ; 
  
 } 
  
 } 
 } 
 

  //go:build examples 
 package 
  
 main 
 import 
  
 ( 
  
 "context" 
  
 dataplex 
  
 "cloud.google.com/go/dataplex/apiv1" 
  
 dataplexpb 
  
 "cloud.google.com/go/dataplex/apiv1/dataplexpb" 
 ) 
 func 
  
 main 
 () 
  
 { 
  
 ctx 
  
 := 
  
 context 
 . 
 Background 
 () 
  
 // This snippet has been automatically generated and should be regarded as a code template only. 
  
 // It will require modifications to work: 
  
 // - It may require correct/in-range values for request initialization. 
  
 // - It may require specifying regional endpoints when creating the service client as shown in: 
  
 //   https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options 
  
 c 
 , 
  
 err 
  
 := 
  
 dataplex 
 . 
  NewCatalogClient 
 
 ( 
 ctx 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 // TODO: Handle error. 
  
 } 
  
 defer 
  
 c 
 . 
 Close 
 () 
  
 req 
  
 := 
  
& dataplexpb 
 . 
 CreateEntryGroupRequest 
 { 
  
 // TODO: Fill request struct fields. 
  
 // See https://pkg.go.dev/cloud.google.com/go/dataplex/apiv1/dataplexpb#CreateEntryGroupRequest. 
  
 } 
  
 op 
 , 
  
 err 
  
 := 
  
 c 
 . 
 CreateEntryGroup 
 ( 
 ctx 
 , 
  
 req 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 // TODO: Handle error. 
  
 } 
  
 resp 
 , 
  
 err 
  
 := 
  
 op 
 . 
 Wait 
 ( 
 ctx 
 ) 
  
 if 
  
 err 
  
 != 
  
 nil 
  
 { 
  
 // TODO: Handle error. 
  
 } 
  
 // TODO: Use resp. 
  
 _ 
  
 = 
  
 resp 
 } 
 

  import 
  
 com.google.cloud.dataplex.v1. CatalogServiceClient 
 
 ; 
 import 
  
 com.google.cloud.dataplex.v1. CreateEntryGroupRequest 
 
 ; 
 import 
  
 com.google.cloud.dataplex.v1. EntryGroup 
 
 ; 
 import 
  
 com.google.cloud.dataplex.v1. LocationName 
 
 ; 
 public 
  
 class 
 SyncCreateEntryGroup 
  
 { 
  
 public 
  
 static 
  
 void 
  
 main 
 ( 
 String 
 [] 
  
 args 
 ) 
  
 throws 
  
 Exception 
  
 { 
  
 syncCreateEntryGroup 
 (); 
  
 } 
  
 public 
  
 static 
  
 void 
  
 syncCreateEntryGroup 
 () 
  
 throws 
  
 Exception 
  
 { 
  
 // This snippet has been automatically generated and should be regarded as a code template only. 
  
 // It will require modifications to work: 
  
 // - It may require correct/in-range values for request initialization. 
  
 // - It may require specifying regional endpoints when creating the service client as shown in 
  
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library 
  
 try 
  
 ( 
  CatalogServiceClient 
 
  
 catalogServiceClient 
  
 = 
  
  CatalogServiceClient 
 
 . 
 create 
 ()) 
  
 { 
  
  CreateEntryGroupRequest 
 
  
 request 
  
 = 
  
  CreateEntryGroupRequest 
 
 . 
 newBuilder 
 () 
  
 . 
 setParent 
 ( 
  LocationName 
 
 . 
 of 
 ( 
 "[PROJECT]" 
 , 
  
 "[LOCATION]" 
 ). 
 toString 
 ()) 
  
 . 
  setEntryGroupId 
 
 ( 
 "entryGroupId1228924712" 
 ) 
  
 . 
 setEntryGroup 
 ( 
  EntryGroup 
 
 . 
 newBuilder 
 (). 
 build 
 ()) 
  
 . 
 setValidateOnly 
 ( 
 true 
 ) 
  
 . 
 build 
 (); 
  
  EntryGroup 
 
  
 response 
  
 = 
  
 catalogServiceClient 
 . 
  createEntryGroupAsync 
 
 ( 
 request 
 ). 
 get 
 (); 
  
 } 
  
 } 
 } 
 

  // Copyright 2026 Google LLC 
 // 
 // Licensed under the Apache License, Version 2.0 (the "License"); 
 // you may not use this file except in compliance with the License. 
 // You may obtain a copy of the License at 
 // 
 //     https://www.apache.org/licenses/LICENSE-2.0 
 // 
 // Unless required by applicable law or agreed to in writing, software 
 // distributed under the License is distributed on an "AS IS" BASIS, 
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
 // See the License for the specific language governing permissions and 
 // limitations under the License. 
 // 
 // ** This file is automatically generated by gapic-generator-typescript. ** 
 // ** https://github.com/googleapis/gapic-generator-typescript ** 
 // ** All changes to this file may be overwritten. ** 
 'use strict' 
 ; 
 function 
  
 main 
 ( 
 parent 
 , 
  
 entryGroupId 
 , 
  
 entryGroup 
 ) 
  
 { 
  
 /** 
 * This snippet has been automatically generated and should be regarded as a code template only. 
 * It will require modifications to work. 
 * It may require correct/in-range values for request initialization. 
 * TODO(developer): Uncomment these variables before running the sample. 
 */ 
  
 /** 
 *  Required. The resource name of the entryGroup, of the form: 
 *  projects/{project_number}/locations/{location_id} 
 *  where `location_id` refers to a Google Cloud region. 
 */ 
  
 // const parent = 'abc123' 
  
 /** 
 *  Required. EntryGroup identifier. 
 */ 
  
 // const entryGroupId = 'abc123' 
  
 /** 
 *  Required. EntryGroup Resource. 
 */ 
  
 // const entryGroup = {} 
  
 /** 
 *  Optional. The service validates the request without performing any 
 *  mutations. The default is false. 
 */ 
  
 // const validateOnly = true 
  
 // Imports the Dataplex library 
  
 const 
  
 { 
 CatalogServiceClient 
 } 
  
 = 
  
 require 
 ( 
 ' @google-cloud/dataplex 
' 
 ). 
 v1 
 ; 
  
 // Instantiates a client 
  
 const 
  
 dataplexClient 
  
 = 
  
 new 
  
  CatalogServiceClient 
 
 (); 
  
 async 
  
 function 
  
 callCreateEntryGroup 
 () 
  
 { 
  
 // Construct request 
  
 const 
  
 request 
  
 = 
  
 { 
  
 parent 
 , 
  
 entryGroupId 
 , 
  
 entryGroup 
 , 
  
 }; 
  
 // Run request 
  
 const 
  
 [ 
 operation 
 ] 
  
 = 
  
 await 
  
 dataplexClient 
 . 
 createEntryGroup 
 ( 
 request 
 ); 
  
 const 
  
 [ 
 response 
 ] 
  
 = 
  
 await 
  
 operation 
 . 
 promise 
 (); 
  
 console 
 . 
 log 
 ( 
 response 
 ); 
  
 } 
  
 callCreateEntryGroup 
 (); 
 } 
 process 
 . 
 on 
 ( 
 'unhandledRejection' 
 , 
  
 err 
  
 = 
>  
 { 
  
 console 
 . 
 error 
 ( 
 err 
 . 
 message 
 ); 
  
 process 
 . 
 exitCode 
  
 = 
  
 1 
 ; 
 }); 
 main 
 (... 
 process 
 . 
 argv 
 . 
 slice 
 ( 
 2 
 ));