Dataflow API Connector Overview

The Workflows connector defines the built-in functions that can be used to access other Google Cloud products within a workflow.

This page provides an overview of the individual connector. There is no need to import or load connector libraries in a workflow—connectors work out of the box when used in a call step.

Dataflow API

Manages Google Cloud Dataflow projects on Google Cloud Platform. To learn more, see the Dataflow API documentation .

Dataflow connector sample

YAML

  # This workflow demonstrates how to use the Cloud Dataflow connector. 
 # The workflow creates a word count job using a Dataflow public job template 
 # and uses a Cloud Storage bucket as temporary storage for temp files. 
 # The bucket resource is deleted after the job completes. 
 # Expected successful output: "SUCCESS" 
 - 
  
 init 
 : 
  
 assign 
 : 
  
 - 
  
 project_id 
 : 
  
 ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")} 
  
 - 
  
 location 
 : 
  
 "us-central1" 
  
 - 
  
 zone 
 : 
  
 "us-central1-a" 
  
 - 
  
 bucket_name 
 : 
  
 "[fill 
  
 in 
  
 a 
  
 bucket 
  
 name]" 
  
 - 
  
 job_name 
 : 
  
 "[fill 
  
 in 
  
 a 
  
 job 
  
 name]" 
  
 - 
  
 input_file 
 : 
  
 "gs://dataflow-samples/shakespeare/kinglear.txt" 
  
 - 
  
 output_storage_file_prefix 
 : 
  
 ${"gs://" + bucket_name + "/counts"} 
  
 - 
  
 temp_location 
 : 
  
 ${"gs://" + bucket_name + "/counts/temp"} 
  
 - 
  
 template_path 
 : 
  
 "gs://dataflow-templates-us-central1/latest/Word_Count" 
 - 
  
 create_bucket 
 : 
  
 call 
 : 
  
 googleapis.storage.v1.buckets.insert 
  
 args 
 : 
  
 project 
 : 
  
 ${project_id} 
  
 body 
 : 
  
 name 
 : 
  
 ${bucket_name} 
 - 
  
 create_job 
 : 
  
 call 
 : 
  
 googleapis.dataflow.v1b3.projects.locations.templates.create 
  
 args 
 : 
  
 projectId 
 : 
  
 ${project_id} 
  
 location 
 : 
  
 ${location} 
  
 body 
 : 
  
 jobName 
 : 
  
 ${job_name} 
  
 parameters 
 : 
  
 inputFile 
 : 
  
 ${input_file} 
  
 output 
 : 
  
 ${output_storage_file_prefix} 
  
 environment 
 : 
  
 numWorkers 
 : 
  
 1 
  
 maxWorkers 
 : 
  
 2 
  
 zone 
 : 
  
 ${zone} 
  
 tempLocation 
 : 
  
 ${temp_location} 
  
 gcsPath 
 : 
  
 ${template_path} 
 - 
  
 delete_bucket_object1 
 : 
  
 call 
 : 
  
 googleapis.storage.v1.objects.delete 
  
 args 
 : 
  
 bucket 
 : 
  
 ${bucket_name} 
  
 object 
 : 
  
 ${"counts-00000-of-00003"} 
 - 
  
 delete_bucket_object2 
 : 
  
 call 
 : 
  
 googleapis.storage.v1.objects.delete 
  
 args 
 : 
  
 bucket 
 : 
  
 ${bucket_name} 
  
 object 
 : 
  
 ${"counts-00001-of-00003"} 
 - 
  
 delete_bucket_object3 
 : 
  
 call 
 : 
  
 googleapis.storage.v1.objects.delete 
  
 args 
 : 
  
 bucket 
 : 
  
 ${bucket_name} 
  
 object 
 : 
  
 ${"counts-00002-of-00003"} 
 - 
  
 delete_bucket 
 : 
  
 call 
 : 
  
 googleapis.storage.v1.buckets.delete 
  
 args 
 : 
  
 bucket 
 : 
  
 ${bucket_name} 
 - 
  
 the_end 
 : 
  
 return 
 : 
  
 "SUCCESS"

JSON

  [ 
  
 { 
  
 "init" 
 : 
  
 { 
  
 "assign" 
 : 
  
 [ 
  
 { 
  
 "project_id" 
 : 
  
 "${sys.get_env(\"GOOGLE_CLOUD_PROJECT_ID\")}" 
  
 }, 
  
 { 
  
 "location" 
 : 
  
 "us-central1" 
  
 }, 
  
 { 
  
 "zone" 
 : 
  
 "us-central1-a" 
  
 }, 
  
 { 
  
 "bucket_name" 
 : 
  
 "[fill in a bucket name]" 
  
 }, 
  
 { 
  
 "job_name" 
 : 
  
 "[fill in a job name]" 
  
 }, 
  
 { 
  
 "input_file" 
 : 
  
 "gs://dataflow-samples/shakespeare/kinglear.txt" 
  
 }, 
  
 { 
  
 "output_storage_file_prefix" 
 : 
  
 "${\"gs://\" + bucket_name + \"/counts\"}" 
  
 }, 
  
 { 
  
 "temp_location" 
 : 
  
 "${\"gs://\" + bucket_name + \"/counts/temp\"}" 
  
 }, 
  
 { 
  
 "template_path" 
 : 
  
 "gs://dataflow-templates-us-central1/latest/Word_Count" 
  
 } 
  
 ] 
  
 } 
  
 }, 
  
 { 
  
 "create_bucket" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.storage.v1.buckets.insert" 
 , 
  
 "args" 
 : 
  
 { 
  
 "project" 
 : 
  
 "${project_id}" 
 , 
  
 "body" 
 : 
  
 { 
  
 "name" 
 : 
  
 "${bucket_name}" 
  
 } 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "create_job" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.dataflow.v1b3.projects.locations.templates.create" 
 , 
  
 "args" 
 : 
  
 { 
  
 "projectId" 
 : 
  
 "${project_id}" 
 , 
  
 "location" 
 : 
  
 "${location}" 
 , 
  
 "body" 
 : 
  
 { 
  
 "jobName" 
 : 
  
 "${job_name}" 
 , 
  
 "parameters" 
 : 
  
 { 
  
 "inputFile" 
 : 
  
 "${input_file}" 
 , 
  
 "output" 
 : 
  
 "${output_storage_file_prefix}" 
  
 }, 
  
 "environment" 
 : 
  
 { 
  
 "numWorkers" 
 : 
  
 1 
 , 
  
 "maxWorkers" 
 : 
  
 2 
 , 
  
 "zone" 
 : 
  
 "${zone}" 
 , 
  
 "tempLocation" 
 : 
  
 "${temp_location}" 
  
 }, 
  
 "gcsPath" 
 : 
  
 "${template_path}" 
  
 } 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "delete_bucket_object1" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.storage.v1.objects.delete" 
 , 
  
 "args" 
 : 
  
 { 
  
 "bucket" 
 : 
  
 "${bucket_name}" 
 , 
  
 "object" 
 : 
  
 "${\"counts-00000-of-00003\"}" 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "delete_bucket_object2" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.storage.v1.objects.delete" 
 , 
  
 "args" 
 : 
  
 { 
  
 "bucket" 
 : 
  
 "${bucket_name}" 
 , 
  
 "object" 
 : 
  
 "${\"counts-00001-of-00003\"}" 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "delete_bucket_object3" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.storage.v1.objects.delete" 
 , 
  
 "args" 
 : 
  
 { 
  
 "bucket" 
 : 
  
 "${bucket_name}" 
 , 
  
 "object" 
 : 
  
 "${\"counts-00002-of-00003\"}" 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "delete_bucket" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.storage.v1.buckets.delete" 
 , 
  
 "args" 
 : 
  
 { 
  
 "bucket" 
 : 
  
 "${bucket_name}" 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "the_end" 
 : 
  
 { 
  
 "return" 
 : 
  
 "SUCCESS" 
  
 } 
  
 } 
 ]

Module: googleapis.dataflow.v1b3.projects.jobs

Functions
`aggregated`	List the jobs of a project across all regions.
`create`	Creates a Cloud Dataflow job. To create a job, we recommend using `projects.locations.jobs.create` with a regional endpoint . Using `projects.jobs.create` is not recommended, as your job will always start in `us-central1` .
`get`	Gets the state of the specified Cloud Dataflow job. To get the state of a job, we recommend using `projects.locations.jobs.get` with a regional endpoint . Using `projects.jobs.get` is not recommended, as you can only get the state of jobs that are running in `us-central1` .
`getMetrics`	Request the job status. To request the status of a job, we recommend using `projects.locations.jobs.getMetrics` with a regional endpoint . Using `projects.jobs.getMetrics` is not recommended, as you can only request the status of jobs that are running in `us-central1` .
`list`	List the jobs of a project. To list the jobs of a project in a region, we recommend using `projects.locations.jobs.list` with a regional endpoint . To list the all jobs across all regions, use `projects.jobs.aggregated` . Using `projects.jobs.list` is not recommended, as you can only get the list of jobs that are running in `us-central1` .
`snapshot`	Snapshot the state of a streaming job.
`update`	Updates the state of an existing Cloud Dataflow job. To update the state of an existing job, we recommend using `projects.locations.jobs.update` with a regional endpoint . Using `projects.jobs.update` is not recommended, as you can only update the state of jobs that are running in `us-central1` .

Module: googleapis.dataflow.v1b3.projects.jobs.messages

Functions
`list`	Request the job status. To request the status of a job, we recommend using `projects.locations.jobs.messages.list` with a regional endpoint . Using `projects.jobs.messages.list` is not recommended, as you can only request the status of jobs that are running in `us-central1` .

Module: googleapis.dataflow.v1b3.projects.locations.flexTemplates

Functions
`launch`	Launch a job with a FlexTemplate.

Module: googleapis.dataflow.v1b3.projects.locations.jobs

Functions
`create`	Creates a Cloud Dataflow job. To create a job, we recommend using `projects.locations.jobs.create` with a regional endpoint . Using `projects.jobs.create` is not recommended, as your job will always start in `us-central1` .
`get`	Gets the state of the specified Cloud Dataflow job. To get the state of a job, we recommend using `projects.locations.jobs.get` with a regional endpoint . Using `projects.jobs.get` is not recommended, as you can only get the state of jobs that are running in `us-central1` .
`getExecutionDetails`	Request detailed information about the execution status of the job. EXPERIMENTAL. This API is subject to change or removal without notice.
`getMetrics`	Request the job status. To request the status of a job, we recommend using `projects.locations.jobs.getMetrics` with a regional endpoint . Using `projects.jobs.getMetrics` is not recommended, as you can only request the status of jobs that are running in `us-central1` .
`list`	List the jobs of a project. To list the jobs of a project in a region, we recommend using `projects.locations.jobs.list` with a regional endpoint . To list the all jobs across all regions, use `projects.jobs.aggregated` . Using `projects.jobs.list` is not recommended, as you can only get the list of jobs that are running in `us-central1` .
`snapshot`	Snapshot the state of a streaming job.
`update`	Updates the state of an existing Cloud Dataflow job. To update the state of an existing job, we recommend using `projects.locations.jobs.update` with a regional endpoint . Using `projects.jobs.update` is not recommended, as you can only update the state of jobs that are running in `us-central1` .

Module: googleapis.dataflow.v1b3.projects.locations.jobs.messages

Functions
`list`	Request the job status. To request the status of a job, we recommend using `projects.locations.jobs.messages.list` with a regional endpoint . Using `projects.jobs.messages.list` is not recommended, as you can only request the status of jobs that are running in `us-central1` .

Module: googleapis.dataflow.v1b3.projects.locations.jobs.stages

Functions
`getExecutionDetails`	Request detailed information about the execution status of a stage of the job. EXPERIMENTAL. This API is subject to change or removal without notice.

Module: googleapis.dataflow.v1b3.projects.locations.templates

Functions
`create`	Creates a Cloud Dataflow job from a template.
`get`	Get the template associated with a template.
`launch`	Launch a template.

Module: googleapis.dataflow.v1b3.projects.templates

Functions
`create`	Creates a Cloud Dataflow job from a template.
`get`	Get the template associated with a template.
`launch`	Launch a template.

Dataflow API Connector Overview Stay organized with collections Save and categorize content based on your preferences.