Console
    Start free

    Monitor Dataplex Universal Catalog logs

    Dataplex Universal Catalog job logs can be viewed, searched, filtered, and archived in Cloud Logging .

    To understand the costs, see Google Cloud Observability pricing .

    For more information about logging retention, see Logs retention periods .

    To disable all logs or exclude logs from Logging, see Exclusion filters .

    To route logs from Logging to Cloud Storage, BigQuery, or Pub/Sub, see Routing and storage overview .

    Access Dataplex Universal Catalog service logs in Logging

    Dataplex Universal Catalog publishes the following service logs to Cloud Logging.

    Log type Log name logName query Log description
    Data scan event logs
    		 dataplex.googleapis.com/data_scan logName=(projects/ $PROJECT_ID /logs/dataplex.googleapis.com%2Fdata_scan) Event logs for data scan jobs indicating job state, results, and statistics
    Data quality scan rule result logs
    		 dataplex.googleapis.com/data_quality_scan_rule_result logName=(projects/ $PROJECT_ID /logs/dataplex.googleapis.com%2Fdata_quality_scan_rule_result) Results of data quality scan rules in a data quality job
    Discovery logs
    		 dataplex.googleapis.com/discovery logName=(projects/ $PROJECT_ID /logs/dataplex.googleapis.com%2Fdiscovery) Discovery progress and updates over assets in a zone
    Metadata job logs
    		 dataplex.googleapis.com/metadata_job logName=(projects/ $PROJECT_ID /logs/dataplex.googleapis.com%2Fmetadata_job) Logs about metadata import jobs and import items in the metadata import file
    Process logs
    		 dataplex.googleapis.com/process logName=(projects/ $PROJECT_ID /logs/dataplex.googleapis.com%2Fprocess) Job runs resulting from data processing tasks

    Replace the following:

    • PROJECT_ID : the ID of your project

    To access Logging, you can use the Logs Explorer in the Google Cloud console, the gcloud logging commands , or the Logging API .

    Query data scan event logs

    When you use Dataplex Universal Catalog to create and run a data scan , a data scan event log is produced in Logging for the resulting job.

    Console

    1. In the Google Cloud console, go to the Logs explorerpage.

      Go to Logs explorer

    2. In the Logs Explorerview, find the Querytab.

    3. Click the Resourcemenu.

    4. Select Cloud Dataplex DataScan. Click Apply.

    5. Click the Log namemenu.

    6. In the Search log namesfield, enter dataplex.googleapis.com%2Fdata_scan . Select data_scanand click Apply.

    7. Optional: Filter the logs to a specific data scan ID or location by adding the following filters in the log query:

      resource.labels.location=" LOCATION 
"
resource.labels.datascan_id=" DATA_SCAN_ID 
"

    8. Click Run query.

    gcloud

    To read your data scan event log entries, use the gcloud logging read command with the following query:

    gcloud logging read \
    'resource.type="dataplex.googleapis.com/DataScan" AND
    logName=projects/ PROJECT_ID 
/logs/dataplex.googleapis.com%2Fdata_scan AND
    resource.labels.location= LOCATION 
AND
    resource.labels.datascan_id= DATA_SCAN_ID 
'
    --limit 10

    REST

    To list log entries, use the entries.list method .

    Query data quality scan rule result logs

    When you use Dataplex Universal Catalog to create and run a data quality scan , a data quality scan rule result log is produced in Logging for the resulting job.

    Console

    1. In the Google Cloud console, go to the Logs explorerpage.

      Go to Logs explorer

    2. In the Logs Explorerview, find the Querytab.

    3. Click the Resourcemenu.

    4. Select Cloud Dataplex DataScan. Click Apply.

    5. Click the Log namemenu.

    6. In the Search log namesfield, enter dataplex.googleapis.com%2Fdata_quality_scan_rule_result . Select data_quality_scan_rule_resultand click Apply.

    7. Optional: Filter the logs to a specific data scan ID or location by adding the following filters in the log query:

      resource.labels.location=" LOCATION 
"
resource.labels.datascan_id=" DATA_SCAN_ID 
"

    8. Click Run query.

    gcloud

    To read your data quality scan rule result log entries, use the gcloud logging read command with the following query:

    gcloud logging read \
    'resource.type="dataplex.googleapis.com/DataScan" AND
    logName=projects/ PROJECT_ID 
/logs/dataplex.googleapis.com%2Fdata_quality_scan_rule_result AND
    resource.labels.location= LOCATION 
AND
    resource.labels.datascan_id= DATA_SCAN_ID 
'
    --limit 10

    REST

    To list log entries, use the entries.list method .

    Query discovery logs

    When you use Dataplex Universal Catalog to discover data in assets, a discovery log is produced in Logging.

    Console

    1. In the Google Cloud console, go to the Logs explorerpage.

      Go to Logs explorer

    2. In the Logs Explorerview, find the Querytab.

    3. Click the Resourcemenu.

    4. Select Cloud Dataplex Zone. Click Apply.

    5. Click the Log namemenu.

    6. In the Search log namesfield, enter dataplex.googleapis.com%2Fdiscovery . Select discoveryand click Apply.

    7. Optional: Filter the logs to a specific asset by adding the following filters in the log query:

      resource.labels.location=" LOCATION 
"
resource.labels.lake_id=" LAKE_ID 
"
resource.labels.zone_id=" ZONE_ID 
"
jsonPayload.assetId=" ASSET_ID 
"

    8. Click Run query.

    gcloud

    To read your discovery log entries, use the gcloud logging read command with the following query:

    gcloud logging read \
    'resource.type="dataplex.googleapis.com/Zone" AND
    logName=projects/ PROJECT_ID 
/logs/dataplex.googleapis.com%2Fdiscovery AND
    resource.labels.location= LOCATION 
AND
    resource.labels.lake_id= LAKE_ID 
AND
    resource.labels.zone_id= ZONE_ID 
AND
    jsonPayload.assetId= ASSET_ID 
'
    --limit 10

    REST

    To list log entries, use the entries.list method .

    Query metadata job logs

    When you run a metadata import job , metadata job logs are produced in Logging.

    Console

    1. In the Google Cloud console, go to the Logs explorerpage.

      Go to Logs explorer

    2. In the Logs Explorerview, find the Querytab.

    3. Click the Resourcemenu.

    4. Select Cloud Dataplex Metadata Job.

    5. Optional: To filter the logs to a specific location or metadata job ID, select a location or job ID.

    6. Click Apply.

    7. Click the Log namemenu.

    8. Type dataplex.googleapis.com%2Fmetadata_job and then select metadata_job.

    9. Click Apply.

    gcloud

    To read your metadata job log entries, use the gcloud logging read command with the following query:

    gcloud logging read \
    'resource.type="dataplex.googleapis.com/MetadataJob" AND
    logName=projects/ PROJECT_ID 
/logs/dataplex.googleapis.com%2Fmetadata_job AND
    resource.labels.location= LOCATION 
AND
    resource.labels.metadata_job_id= METADATA_JOB_ID 
--limit 10

    REST

    To list log entries, use the entries.list method .

    Query process logs

    When you use Dataplex Universal Catalog to schedule and run tasks , a process log is produced in Logging for the resulting job.

    Console

    1. In the Google Cloud console, go to the Logs explorerpage.

      Go to Logs explorer

    2. In the Logs Explorerview, find the Querytab.

    3. Click the Resourcemenu.

    4. Select Cloud Dataplex Task. Click Apply.

    5. Click the Log namemenu.

    6. In the Search log namesfield, enter dataplex.googleapis.com%2Fprocess . Select processand click Apply.

    7. Optional: Filter the logs to a specific task by adding the following filters in the log query:

      resource.labels.location=" LOCATION 
"
resource.labels.lake_id=" LAKE_ID 
"
resource.labels.task_id=" TASK_ID 
"

    8. Click Run query.

    gcloud

    To read your process log entries, use the gcloud logging read command with the following query:

    gcloud logging read \
    'resource.type="dataplex.googleapis.com/Task" AND
    logName=projects/ PROJECT_ID 
/logs/dataplex.googleapis.com%2Fprocess AND
    resource.labels.location= LOCATION 
AND
    resource.labels.lake_id= LAKE_ID 
AND
    resource.labels.task_id= TASK_ID 
'
    --limit 10

    REST

    To list log entries, use the entries.list method .

    What's next
    Create a Mobile Website
    View Site in Mobile | Classic
    Share by: