Manage metadata of lakes, zones, and assets

List entities

To limit the list of entities returned by the service, add filter query parameters to the list entities request URL.

Get entity

By default, the Get Entity response contains basic entity metadata. To retrieve additional schema metadata, add the view query parameter to the request URL.

Compatibility details:While Dataplex Universal Catalog metadata is centrally registered in the metadata API, only entity table metadata that is compatible with BigQuery and Apache Hive Metastore is published to BigQuery and Dataproc Metastore. The Get Entity API returns a CompatibilityStatus message, which indicates if table metadata is compatible with BigQuery and Hive Metastore, and if not, the reason for the incompatibility.

Update entity

Use this API to edit entity metadata, including whether you or Dataplex Universal Catalog will manage entity metadata.

• This API performs a full replacement of all mutable Entity fields. The following Entity fields are immutable, and if you specify them in an update request, they will be ignored:
  • asset
  • dataPath
  • type
  • system
• Specify a value for all mutable Entity fields, including all schema fields, even if the values are not being changed.
• Supply the etag field. You can obtain the etag by first submitting a entities.get request, which returns the etag of the entity in the response.
• Updating schema fields:You can update the table schema discovered by Dataplex Universal Catalog to improve its accuracy:
  • If the schema is a fileset, leave all schema fields empty.
  • To define a repeated field, set the mode to REPEATED . To define a struct field, set the type to RECORD .
  • You can set the userManaged field of the schema to specify whether you or Dataplex Universal Catalog manages table metadata. The default setting is Dataplex Universal Catalog managed. If userManaged is set to true, this setting is included in the information returned from an entities.get request if EntityView is set to SCHEMA or FULL .
• Updating partition fields:
  • For non-Hive style partitioned data, Dataplex Universal Catalog discovery auto-generates partition keys. For example, for the data path gs://root/2020/12/31 , partition keys p0 , p1 , and p2 are generated. To make querying more intuitive, you can update p0 , p1 , and p2 to year , month , and day respectively.
  • If you update the partition style to HIVE style , the partition field is immutable.
• Updating other metadata fields:You can update auto-generated mimeType , CompressionFormat , CsvOptions , and JsonOptions fields to aid Dataplex Universal Catalog discovery. Dataplex Universal Catalog discovery will use new values on its next run.

Create entity

Use the entities.create API to create table or fileset metadata entities. Populate the required and relevant optional fields, or let the Dataplex Universal Catalog discovery service fill in optional fields.

Delete entity

• Supply the etag field. You can obtain the etag by first submitting a entities.get request, which returns the etag of the entity in the response.

If underlying data for a table or fileset in a raw zone is deleted, the table or fileset metadata get deleted automatically upon the next Discovery scan. If underlying data for a table in a curated zone is deleted, the table metadata isn't deleted correspondingly, but rather, a missing data action is reported. To resolve this issue, explicitly delete the table metadata entity through the metadata API.

List partitions

To limit the list of partitions returned by the service, add filter query parameters to the list partitions request URL.

Examples:

• ?filter="Country=US AND State=CA AND City=Sunnyvale"
• ?filter="year < 2000 AND month > 12 AND Date > 10"

Get partition

To get a partition, you must complete the request URL by appending the partition key values to the end of the URL, formatted to read as partitions/value1/value2/…./value10 .

Example: if a partition has values, {Country=US, State=CA, City=Sunnyvale} , the get request URL should end with /partitions/US/CA/Sunnyvale .

Important:The appended URL values must be double encoded. For example, url_encode(url_encode(value)) can be used to encode "US:CA/CA#Sunnyvale" so that the request URL ends with /partitions/US%253ACA/CA%2523Sunnyvale . The name field in the response retains the encoded format.

Create partition

To create a customized partition for your data source, use the partitions.create API. Specify the required location field with a Cloud Storage path.

Delete partition

Complete the request URL by appending partition key values to the end of the request URL, formatted to read as partitions/value1/value2/…./value10 .

Example: if a partition has values, {Country=US, State=CA, City=Sunnyvale} , the request URL should end with /partitions/US/CA/Sunnyvale .

Important:The appended URL values must conform to RFC-1034 or they must be double encoded, for example, US:/CA#/Sunnyvale as US%3A/CA%3A/Sunnyvale .