Console
    Contact Us Start free

    REST Resource: projects.locations.workflowTemplates

    Resource: WorkflowTemplate

    A Dataproc workflow template resource.

    JSON representation 
     { 
 "id" 
 : 
 string 
 , 
 "name" 
 : 
 string 
 , 
 "version" 
 : 
 integer 
 , 
 "createTime" 
 : 
 string 
 , 
 "updateTime" 
 : 
 string 
 , 
 "labels" 
 : 
 { 
 string 
 : 
 string 
 , 
 ... 
 } 
 , 
 "placement" 
 : 
 { 
 object (  WorkflowTemplatePlacement 
 
) 
 } 
 , 
 "jobs" 
 : 
 [ 
 { 
 object (  OrderedJob 
 
) 
 } 
 ] 
 , 
 "parameters" 
 : 
 [ 
 { 
 object (  TemplateParameter 
 
) 
 } 
 ] 
 , 
 "dagTimeout" 
 : 
 string 
 , 
 "encryptionConfig" 
 : 
 { 
 object (  EncryptionConfig 
 
) 
 } 
 }
    Fields
    id

    string

    name

    string

    Output only. The resource name of the workflow template, as described in https://cloud.google.com/apis/design/resource_names .

    • For projects.regions.workflowTemplates , the resource name of the template has the following format: projects/{projectId}/regions/{region}/workflowTemplates/{template_id}

    • For projects.locations.workflowTemplates , the resource name of the template has the following format: projects/{projectId}/locations/{location}/workflowTemplates/{template_id}

    version

    integer

    Optional. Used to perform a consistent read-modify-write.

    This field should be left blank for a workflowTemplates.create request. It is required for an workflowTemplates.update request, and must match the current server version. A typical update template flow would fetch the current template with a workflowTemplates.get request, which will return the current template with the version field filled in with the current server version. The user updates other fields in the template, then returns it as part of the workflowTemplates.update request.

    createTime

    string ( Timestamp format)

    Output only. The time template was created.

    Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z" , "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30" .

    updateTime

    string ( Timestamp format)

    Output only. The time template was last updated.

    Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z" , "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30" .

    labels

    map (key: string, value: string)

    Optional. The labels to associate with this template. These labels will be propagated to all jobs and clusters created by the workflow instance.

    Label keysmust contain 1 to 63 characters, and must conform to RFC 1035 .

    Label valuesmay be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 .

    No more than 32 labels can be associated with a template.

    An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" } .

    placement

    object ( WorkflowTemplatePlacement )

    Required. WorkflowTemplate scheduling information.

    jobs[]

    object ( OrderedJob )

    Required. The Directed Acyclic Graph of Jobs to submit.

    parameters[]

    object ( TemplateParameter )

    Optional. Template parameters whose values are substituted into the template. Values for parameters must be provided when the template is instantiated.

    dagTimeout

    string ( Duration format)

    Optional. Timeout duration for the DAG of jobs, expressed in seconds (see JSON representation of duration ). The timeout duration must be from 10 minutes ("600s") to 24 hours ("86400s"). The timer begins when the first job is submitted. If the workflow is running at the end of the timeout period, any remaining jobs are cancelled, the workflow is ended, and if the workflow was running on a managed cluster , the cluster is deleted.

    encryptionConfig

    object ( EncryptionConfig )

    Optional. Encryption settings for encrypting workflow template job arguments.

    WorkflowTemplatePlacement

    Specifies workflow execution target.

    Either managedCluster or clusterSelector is required.

    JSON representation 
     { 
 // Union field placement 
can be only one of the following: 
 "managedCluster" 
 : 
 { 
 object (  ManagedCluster 
 
) 
 } 
 , 
 "clusterSelector" 
 : 
 { 
 object (  ClusterSelector 
 
) 
 } 
 // End of list of possible types for union field placement 
. 
 }
    Fields
    Union field placement . Required. Specifies where workflow executes; either on a managed cluster or an existing cluster chosen by labels. placement can be only one of the following:
    managedCluster

    object ( ManagedCluster )

    A cluster that is managed by the workflow.

    clusterSelector

    object ( ClusterSelector )

    Optional. A selector that chooses target cluster for jobs based on metadata.

    The selector is evaluated at the time each job is submitted.

    ManagedCluster

    Cluster that is managed by the workflow.

    JSON representation 
     { 
 "clusterName" 
 : 
 string 
 , 
 "config" 
 : 
 { 
 object (  ClusterConfig 
 
) 
 } 
 , 
 "labels" 
 : 
 { 
 string 
 : 
 string 
 , 
 ... 
 } 
 }
    Fields
    clusterName

    string

    Required. The cluster name prefix. A unique cluster name will be formed by appending a random suffix.

    The name must contain only lower-case letters (a-z), numbers (0-9), and hyphens (-). Must begin with a letter. Cannot begin or end with hyphen. Must consist of between 2 and 35 characters.
    config

    object ( ClusterConfig )

    Required. The cluster configuration.
    labels

    map (key: string, value: string)

    Optional. The labels to associate with this cluster.

    Label keys must be between 1 and 63 characters long, and must conform to the following PCRE regular expression: [\p{Ll}\p{Lo}][\p{Ll}\p{Lo}\p{N}_-]{0,62}

    Label values must be between 1 and 63 characters long, and must conform to the following PCRE regular expression: [\p{Ll}\p{Lo}\p{N}_-]{0,63}

    No more than 32 labels can be associated with a given cluster.

    An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" } .

    ClusterSelector

    A selector that chooses target cluster for jobs based on metadata.

    JSON representation 
     { 
 "zone" 
 : 
 string 
 , 
 "clusterLabels" 
 : 
 { 
 string 
 : 
 string 
 , 
 ... 
 } 
 }
    Fields
    zone

    string

    Optional. The zone where workflow process executes. This parameter does not affect the selection of the cluster.

    If unspecified, the zone of the first cluster matching the selector is used.
    clusterLabels

    map (key: string, value: string)

    Required. The cluster labels. Cluster must have all labels to match.

    An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" } .

    OrderedJob

    A job executed by the workflow.

    JSON representation 
     { 
 "stepId" 
 : 
 string 
 , 
 "labels" 
 : 
 { 
 string 
 : 
 string 
 , 
 ... 
 } 
 , 
 "scheduling" 
 : 
 { 
 object (  JobScheduling 
 
) 
 } 
 , 
 "prerequisiteStepIds" 
 : 
 [ 
 string 
 ] 
 , 
 // Union field job_type 
can be only one of the following: 
 "hadoopJob" 
 : 
 { 
 object (  HadoopJob 
 
) 
 } 
 , 
 "sparkJob" 
 : 
 { 
 object (  SparkJob 
 
) 
 } 
 , 
 "pysparkJob" 
 : 
 { 
 object (  PySparkJob 
 
) 
 } 
 , 
 "hiveJob" 
 : 
 { 
 object (  HiveJob 
 
) 
 } 
 , 
 "pigJob" 
 : 
 { 
 object (  PigJob 
 
) 
 } 
 , 
 "sparkRJob" 
 : 
 { 
 object (  SparkRJob 
 
) 
 } 
 , 
 "sparkSqlJob" 
 : 
 { 
 object (  SparkSqlJob 
 
) 
 } 
 , 
 "prestoJob" 
 : 
 { 
 object (  PrestoJob 
 
) 
 } 
 , 
 "flinkJob" 
 : 
 { 
 object (  FlinkJob 
 
) 
 } 
 // End of list of possible types for union field job_type 
. 
 }
    Fields
    stepId

    string

    Required. The step id. The id must be unique among all jobs within the template.

    The step id is used as prefix for job id, as job goog-dataproc-workflow-step-id label, and in prerequisiteStepIds field from other steps.

    The id must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), and hyphens (-). Cannot begin or end with underscore or hyphen. Must consist of between 3 and 50 characters.

    labels

    map (key: string, value: string)

    Optional. The labels to associate with this job.

    Label keys must be between 1 and 63 characters long, and must conform to the following regular expression: [\p{Ll}\p{Lo}][\p{Ll}\p{Lo}\p{N}_-]{0,62}

    Label values must be between 1 and 63 characters long, and must conform to the following regular expression: [\p{Ll}\p{Lo}\p{N}_-]{0,63}

    No more than 32 labels can be associated with a given job.

    An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" } .

    scheduling

    object ( JobScheduling )

    Optional. Job scheduling configuration.

    prerequisiteStepIds[]

    string

    Optional. The optional list of prerequisite job step_ids. If not specified, the job will start at the beginning of workflow.

    Union field job_type . Required. The job definition. job_type can be only one of the following:
    hadoopJob

    object ( HadoopJob )

    Optional. Job is a Hadoop job.

    sparkJob

    object ( SparkJob )

    Optional. Job is a Spark job.

    pysparkJob

    object ( PySparkJob )

    Optional. Job is a PySpark job.

    hiveJob

    object ( HiveJob )

    Optional. Job is a Hive job.

    pigJob

    object ( PigJob )

    Optional. Job is a Pig job.

    sparkRJob

    object ( SparkRJob )

    Optional. Job is a SparkR job.

    sparkSqlJob

    object ( SparkSqlJob )

    Optional. Job is a SparkSql job.

    prestoJob

    object ( PrestoJob )

    Optional. Job is a Presto job.

    TemplateParameter

    A configurable parameter that replaces one or more fields in the template. Parameterizable fields: - Labels - File uris - Job properties - Job arguments - Script variables - Main class (in HadoopJob and SparkJob) - Zone (in ClusterSelector)

    JSON representation 
     { 
 "name" 
 : 
 string 
 , 
 "fields" 
 : 
 [ 
 string 
 ] 
 , 
 "description" 
 : 
 string 
 , 
 "validation" 
 : 
 { 
 object (  ParameterValidation 
 
) 
 } 
 }
    Fields
    name

    string

    Required. Parameter name. The parameter name is used as the key, and paired with the parameter value, which are passed to the template when the template is instantiated. The name must contain only capital letters (A-Z), numbers (0-9), and underscores (_), and must not start with a number. The maximum length is 40 characters.

    fields[]

    string

    Required. Paths to all fields that the parameter replaces. A field is allowed to appear in at most one parameter's list of field paths.

    A field path is similar in syntax to a google.protobuf.FieldMask . For example, a field path that references the zone field of a workflow template's cluster selector would be specified as placement.clusterSelector.zone .

    Also, field paths can reference fields using the following syntax:

    • Values in maps can be referenced by key:

      • labels['key']
      • placement.clusterSelector.clusterLabels['key']
      • placement.managedCluster.labels['key']
      • placement.clusterSelector.clusterLabels['key']
      • jobs['step-id'].labels['key']

    • Jobs in the jobs list can be referenced by step-id:

      • jobs['step-id'].hadoopJob.mainJarFileUri
      • jobs['step-id'].hiveJob.queryFileUri
      • jobs['step-id'].pySparkJob.mainPythonFileUri
      • jobs['step-id'].hadoopJob.jarFileUris[0]
      • jobs['step-id'].hadoopJob.archiveUris[0]
      • jobs['step-id'].hadoopJob.fileUris[0]
      • jobs['step-id'].pySparkJob.pythonFileUris[0]

    • Items in repeated fields can be referenced by a zero-based index:

      • jobs['step-id'].sparkJob.args[0]

    • Other examples:

      • jobs['step-id'].hadoopJob.properties['key']
      • jobs['step-id'].hadoopJob.args[0]
      • jobs['step-id'].hiveJob.scriptVariables['key']
      • jobs['step-id'].hadoopJob.mainJarFileUri
      • placement.clusterSelector.zone

    It may not be possible to parameterize maps and repeated fields in their entirety since only individual map values and individual items in repeated fields can be referenced. For example, the following field paths are invalid:

    • placement.clusterSelector.clusterLabels
    • jobs['step-id'].sparkJob.args
    description

    string

    Optional. Brief description of the parameter. Must not exceed 1024 characters.

    validation

    object ( ParameterValidation )

    Optional. Validation rules to be applied to this parameter's value.

    ParameterValidation

    Configuration for parameter validation.

    JSON representation 
     { 
 // Union field validation_type 
can be only one of the following: 
 "regex" 
 : 
 { 
 object (  RegexValidation 
 
) 
 } 
 , 
 "values" 
 : 
 { 
 object (  ValueValidation 
 
) 
 } 
 // End of list of possible types for union field validation_type 
. 
 }
    Fields
    Union field validation_type . Required. The type of validation to be performed. validation_type can be only one of the following:
    regex

    object ( RegexValidation )

    Validation based on regular expressions.

    values

    object ( ValueValidation )

    Validation based on a list of allowed values.

    RegexValidation

    Validation based on regular expressions.

    JSON representation 
     { 
 "regexes" 
 : 
 [ 
 string 
 ] 
 }
    Fields
    regexes[]

    string

    Required. RE2 regular expressions used to validate the parameter's value. The value must match the regex in its entirety (substring matches are not sufficient).

    ValueValidation

    Validation based on a list of allowed values.

    JSON representation 
     { 
 "values" 
 : 
 [ 
 string 
 ] 
 }
    Fields
    values[]

    string

    Required. List of allowed values for the parameter.

    EncryptionConfig

    Encryption settings for encrypting workflow template job arguments.

    JSON representation 
     { 
 "kmsKey" 
 : 
 string 
 }
    Fields
    kmsKey

    string

    Optional. The Cloud KMS key name to use for encrypting workflow template job arguments.

    When this this key is provided, the following workflow template job arguments , if present, are CMEK encrypted :

    Methods

    create

    		Creates new workflow template.

    delete

    		Deletes a workflow template.

    get

    		Retrieves the latest workflow template.

    getIamPolicy

    		Gets the access control policy for a resource.

    instantiate

    		Instantiates a template and begins execution.

    instantiateInline

    		Instantiates a template and begins execution.

    list

    		Lists workflows that match the specified filter in the request.

    setIamPolicy

    		Sets the access control policy on the specified resource.

    testIamPermissions

    		Returns permissions that a caller has on the specified resource.

    update

    		Updates (replaces) workflow template.
    Create a Mobile Website
    View Site in Mobile | Classic
    Share by: