Create pipeline alerts

This page describes how to set up alerts for data pipelines in Cloud Data Fusion.

You can create alerts through the following services:

• Alerts through Cloud Data Fusion: you set up these alerts when you design a batch pipeline in Cloud Data Fusion. After you deploy the pipeline, you can't add or edit an alert.

• Log-based alerts through Cloud Monitoring: you set up these alerts in the Google Cloud console before and after you deploy the pipeline.

Before you begin

You need a Cloud Data Fusion instance. For more information, see Creating a Cloud Data Fusion instance .

Add an alert in Cloud Data Fusion

1. Open your instance.

   1. In the Google Cloud console, go to the Cloud Data Fusion page.

   2. To open the instance in the Cloud Data Fusion Studio, click Instances, and then click View instance.

      Go to Instances

2. Go to the Studiopage where you're designing your pipeline.

3. Click Configure > Pipeline alerts > add Add.

4. Select and configure the alert. For more information about configuring alerts, see Batch pipeline alerts .

5. After you configure the alert, click Validateto check for errors, and then click Confirm.

Create an alert in Monitoring

1. Enable Cloud Logging in your Cloud Data Fusion instance.

2. Create a log-based alert .

   1. In the Google Cloud console, go to the Logs Explorerpage.

      Go to the Logs Explorer

      This page lets you store, query, update, and delete data.

   2. Enter the following query:

       resource.type = 
       "cloud_dataproc_cluster" 
       logName 
       = 
       "projects/ PROJECT_ID 
      /logs/datafusion-pipeline-logs" 
      resource.labels.project_id = 
       " PROJECT_ID 
      " 
       severity 
       = 
      ERROR
      jsonPayload.message = 
      ~ "Pipeline '.*' failed." 
       
      

      Replace PROJECT_ID with your project's ID.

   3. Click Create alertand configure the alert.

An alert is only generated when the pipeline runs on a Dataproc cluster and fails. For Dataproc cluster creation failures, the alert is not generated.

What's next

• Refer to the CDAP OS Batch Pipeline Alerts reference pages.