Create a pipeline monitoring dashboard using Cloud Monitoring

Create a log-based metric

1. Go to the Cloud Logging Log-based metricspage:

   Go to Log-based metrics

2. Click Create metric.

3. On the Create a metricpage, do the following:

   1. For Metric type, select Counter.
   2. In the Log-based metric namefield, enter pipeline_logs .
   3. In the Unitsfield, enter 1 .

   4. In the Build filterfield, enter the following:

       resource.type="cloud_dataproc_cluster"
      log_name=~"projects/.*/logs/datafusion-pipeline-logs" 
      

   5. In the Labelssection, click Add labeland create the following labels. After entering each label, click Done, and click Add labelagain to create the next label.

      Label name	Label type	Field name
      Project	STRING	resource.labels.project_id
      Message	STRING	jsonPayload.message
      LoggerName	STRING	labels.loggerName
      ClusterName	STRING	resource.labels.cluster_name
      SparkPhase	STRING	labels.".workflowSparkId"
      Region	STRING	resource.labels.region
      Pipeline	STRING	labels.".applicationId"
      RunId	STRING	labels.".runId"
      Namespace	STRING	labels.".namespaceId"
      LogLevel	STRING	labels.levelName

   6. Click Create metric.

      The newly created metric appears in the user-defined metrics table. If the metric isn't immediately visible, refresh the page.

      The dashboard contains the following charts:

      • All pipelines
      • Completed pipelines
      • Failed pipelines
      • All pipeline runs
      • Completed pipeline runs
      • Failed pipeline runs
      • Dataproc clusters for runs

      After a metric is created, it might take up to 24 hours to start displaying the time series data.

Install the dashboard

1. Download the JSON file to your local machine.

2. Go to the Cloud Monitoring Dashboardspage:

   Go to Monitoring dashboards

3. Click Create dashboard.

4. Click settings Dashboard settings > JSON > JSON editor.

5. In a text editor, open the JSON file that you downloaded.

6. Copy the content of the downloaded JSON file and paste it into the JSON editor, replacing the content that the JSON editor contains by default.

7. Click Apply changes.

   This refreshes the dashboard. The Cloud Data Fusion pipelines run after the metric was created, appear in the dashboard. If no pipelines were run after the metric was created, the dashboard will be empty.

8. Autosave is enabled by default. If autosave is disabled, click Saveto save the dashboard.

9. Click Close editor.

   Your new dashboard appears in the list of dashboards on the Monitoring overviewpage.