You can view, search, filter, and archive Dataproc Metastore job logs in Cloud Logging .
-
See Google Cloud Observability Pricing to understand your costs.
-
See Logs retention periods for information on logging retention.
-
See Logs Exclusions to disable all logs or exclude logs from Logging.
-
See Routing and storage overview to route logs from Logging to Cloud Storage, BigQuery, or Pub/Sub.
Access job logs in Logging
When you run Dataproc Metastore, a Hive metastore process output is streamed to the Google Cloud console.
You can access Logging using the Logging console , the gcloud logging command, or the Logging API .
Console
-
In the Google Cloud console, go to the Cloud Logging > Logs(Logs Explorer) page:
-
Select an existing Dataproc Metastore Google Cloud project.
-
Under the Resourcesmenu, select
Dataproc Metastore Service. -
Under the All logsmenu, select the log type that you want to see.
gcloud
The Google Cloud CLI has a group of commands, gcloud logging
, that provide a
command-line interface to the Cloud Logging API. To read your log entries,
run the following command:
gcloud logging read \ 'resource.type="metastore.googleapis.com/Service" resource.labels.location= LOCATION resource.labels.service_id= SERVICE_ID '
REST
Follow the Logging API instructions to list log entries ( entries.list ) by using the API Explorer.
Dataproc Metastore service logs in Logging
Dataproc Metastore exports request and system logs to Cloud Logging.
| Log Type | Log Name | logName query |
|---|---|---|
|
Request logs
|
metastore.googleapis.com/requests | logName=(projects/$PROJECT_ID/logs/metastore.googleapis.com%2Frequests) |
|
System logs
|
metastore.googleapis.com/system | logName=(projects/$PROJECT_ID/logs/metastore.googleapis.com%2Fsystem) |
