BigQuery Data Transfer API Connector Overview

The Workflows connector defines the built-in functions that can be used to access other Google Cloud products within a workflow.

This page provides an overview of the individual connector. There is no need to import or load connector libraries in a workflow—connectors work out of the box when used in a call step.

BigQuery Data Transfer API

Schedule queries or transfer external data from SaaS applications to Google BigQuery on a regular basis. To learn more, see the BigQuery Data Transfer API documentation .

BigQuery Data Transfer connector sample

YAML

  # This workflow creates a new dataset and a new table inside that dataset, which are required 
 # for the BigQuery Data Transfer Job to run. It creates a new TransferJob configuration and starts 
 # a manual run of the transfer (30 seconds after the config is created). 
 # The transferRun is a blocking LRO. 
 # All resources get deleted once the transfer run completes. 
 # 
 # On success, it returns "SUCCESS". 
 # 
 # Features included in this test: 
 # - BigQuery Data Transfer connector 
 # - Waiting for long-running transfer run to complete 
 # 
 # This workflow expects following items to be provided through input argument for execution: 
 #   - projectID (string) 
 #     - The user project ID. 
 #   - datasetID (string) 
 #     - The dataset name, expected to have an unique value to avoid the 
 #       instance being referred by multiple tests. 
 #   - tableID (string) 
 #     - The table name, expected to have an unique value to avoid the 
 #       instance being referred by multiple tests. 
 #   - runConfigDisplayName (string) 
 #     - The transfer run configuration display name. 
 # 
 # Expected successful output: "SUCCESS" 
 main 
 : 
  
 params 
 : 
  
 [ 
 args 
 ] 
  
 steps 
 : 
  
 - 
  
 init 
 : 
  
 assign 
 : 
  
 - 
  
 project_id 
 : 
  
 ${args.projectID} 
  
 - 
  
 destination_dataset 
 : 
  
 ${args.datasetID} 
  
 - 
  
 destination_table 
 : 
  
 ${args.tableID} 
  
 - 
  
 run_config_display_name 
 : 
  
 ${args.runConfigDisplayName} 
  
 - 
  
 run_config_data_source_id 
 : 
  
 "google_cloud_storage" 
  
 - 
  
 location 
 : 
  
 "us" 
  
 - 
  
 data_path_template 
 : 
  
 "gs://xxxxxx-bucket/xxxxx/xxxx" 
  
 - 
  
 create_dataset 
 : 
  
 call 
 : 
  
 googleapis.bigquery.v2.datasets.insert 
  
 args 
 : 
  
 projectId 
 : 
  
 ${project_id} 
  
 body 
 : 
  
 datasetReference 
 : 
  
 datasetId 
 : 
  
 ${destination_dataset} 
  
 projectId 
 : 
  
 ${project_id} 
  
 - 
  
 create_table 
 : 
  
 call 
 : 
  
 googleapis.bigquery.v2.tables.insert 
  
 args 
 : 
  
 datasetId 
 : 
  
 ${destination_dataset} 
  
 projectId 
 : 
  
 ${project_id} 
  
 body 
 : 
  
 tableReference 
 : 
  
 datasetId 
 : 
  
 ${destination_dataset} 
  
 projectId 
 : 
  
 ${project_id} 
  
 tableId 
 : 
  
 ${destination_table} 
  
 schema 
 : 
  
 fields 
 : 
  
 - 
  
 name 
 : 
  
 "column1" 
  
 type 
 : 
  
 "STRING" 
  
 - 
  
 name 
 : 
  
 "column2" 
  
 type 
 : 
  
 "STRING" 
  
 - 
  
 list_config 
 : 
  
 call 
 : 
  
 googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.list 
  
 args 
 : 
  
 parent 
 : 
  
 ${"projects/" + project_id + "/locations/us"} 
  
 - 
  
 create_run_config 
 : 
  
 call 
 : 
  
 googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.create 
  
 args 
 : 
  
 parent 
 : 
  
 ${"projects/" + project_id + "/locations/" + location} 
  
 body 
 : 
  
 displayName 
 : 
  
 ${run_config_display_name} 
  
 schedule 
 : 
  
 "every 
  
 day 
  
 19:22" 
  
 scheduleOptions 
 : 
  
 disableAutoScheduling 
 : 
  
 true 
  
 destinationDatasetId 
 : 
  
 ${destination_dataset} 
  
 dataSourceId 
 : 
  
 ${run_config_data_source_id} 
  
 params 
 : 
  
 destination_table_name_template 
 : 
  
 ${destination_table} 
  
 file_format 
 : 
  
 "CSV" 
  
 data_path_template 
 : 
  
 ${data_path_template} 
  
 result 
 : 
  
 config 
  
 - 
  
 get_time_in_30s 
 : 
  
 assign 
 : 
  
 - 
  
 now_plus_30s 
 : 
  
 ${time.format(sys.now() + 30)} 
  
 - 
  
 start_run 
 : 
  
 call 
 : 
  
 googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.startManualRuns 
  
 args 
 : 
  
 parent 
 : 
  
 ${config.name} 
  
 body 
 : 
  
 requestedRunTime 
 : 
  
 ${now_plus_30s} 
  
 result 
 : 
  
 runsResp 
  
 - 
  
 remove_run_config 
 : 
  
 call 
 : 
  
 googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.delete 
  
 args 
 : 
  
 name 
 : 
  
 ${config.name} 
  
 - 
  
 delete_table 
 : 
  
 call 
 : 
  
 googleapis.bigquery.v2.tables.delete 
  
 args 
 : 
  
 datasetId 
 : 
  
 ${destination_dataset} 
  
 projectId 
 : 
  
 ${project_id} 
  
 tableId 
 : 
  
 ${destination_table} 
  
 - 
  
 delete_dataset 
 : 
  
 call 
 : 
  
 googleapis.bigquery.v2.datasets.delete 
  
 args 
 : 
  
 projectId 
 : 
  
 ${project_id} 
  
 datasetId 
 : 
  
 ${destination_dataset} 
  
 - 
  
 the_end 
 : 
  
 return 
 : 
  
 "SUCCESS"

JSON

  { 
  
 "main" 
 : 
  
 { 
  
 "params" 
 : 
  
 [ 
  
 "args" 
  
 ], 
  
 "steps" 
 : 
  
 [ 
  
 { 
  
 "init" 
 : 
  
 { 
  
 "assign" 
 : 
  
 [ 
  
 { 
  
 "project_id" 
 : 
  
 "${args.projectID}" 
  
 }, 
  
 { 
  
 "destination_dataset" 
 : 
  
 "${args.datasetID}" 
  
 }, 
  
 { 
  
 "destination_table" 
 : 
  
 "${args.tableID}" 
  
 }, 
  
 { 
  
 "run_config_display_name" 
 : 
  
 "${args.runConfigDisplayName}" 
  
 }, 
  
 { 
  
 "run_config_data_source_id" 
 : 
  
 "google_cloud_storage" 
  
 }, 
  
 { 
  
 "location" 
 : 
  
 "us" 
  
 }, 
  
 { 
  
 "data_path_template" 
 : 
  
 "gs://xxxxxx-bucket/xxxxx/xxxx" 
  
 } 
  
 ] 
  
 } 
  
 }, 
  
 { 
  
 "create_dataset" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.bigquery.v2.datasets.insert" 
 , 
  
 "args" 
 : 
  
 { 
  
 "projectId" 
 : 
  
 "${project_id}" 
 , 
  
 "body" 
 : 
  
 { 
  
 "datasetReference" 
 : 
  
 { 
  
 "datasetId" 
 : 
  
 "${destination_dataset}" 
 , 
  
 "projectId" 
 : 
  
 "${project_id}" 
  
 } 
  
 } 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "create_table" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.bigquery.v2.tables.insert" 
 , 
  
 "args" 
 : 
  
 { 
  
 "datasetId" 
 : 
  
 "${destination_dataset}" 
 , 
  
 "projectId" 
 : 
  
 "${project_id}" 
 , 
  
 "body" 
 : 
  
 { 
  
 "tableReference" 
 : 
  
 { 
  
 "datasetId" 
 : 
  
 "${destination_dataset}" 
 , 
  
 "projectId" 
 : 
  
 "${project_id}" 
 , 
  
 "tableId" 
 : 
  
 "${destination_table}" 
  
 }, 
  
 "schema" 
 : 
  
 { 
  
 "fields" 
 : 
  
 [ 
  
 { 
  
 "name" 
 : 
  
 "column1" 
 , 
  
 "type" 
 : 
  
 "STRING" 
  
 }, 
  
 { 
  
 "name" 
 : 
  
 "column2" 
 , 
  
 "type" 
 : 
  
 "STRING" 
  
 } 
  
 ] 
  
 } 
  
 } 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "list_config" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.list" 
 , 
  
 "args" 
 : 
  
 { 
  
 "parent" 
 : 
  
 "${\"projects/\" + project_id + \"/locations/us\"}" 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "create_run_config" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.create" 
 , 
  
 "args" 
 : 
  
 { 
  
 "parent" 
 : 
  
 "${\"projects/\" + project_id + \"/locations/\" + location}" 
 , 
  
 "body" 
 : 
  
 { 
  
 "displayName" 
 : 
  
 "${run_config_display_name}" 
 , 
  
 "schedule" 
 : 
  
 "every day 19:22" 
 , 
  
 "scheduleOptions" 
 : 
  
 { 
  
 "disableAutoScheduling" 
 : 
  
 true 
  
 }, 
  
 "destinationDatasetId" 
 : 
  
 "${destination_dataset}" 
 , 
  
 "dataSourceId" 
 : 
  
 "${run_config_data_source_id}" 
 , 
  
 "params" 
 : 
  
 { 
  
 "destination_table_name_template" 
 : 
  
 "${destination_table}" 
 , 
  
 "file_format" 
 : 
  
 "CSV" 
 , 
  
 "data_path_template" 
 : 
  
 "${data_path_template}" 
  
 } 
  
 } 
  
 }, 
  
 "result" 
 : 
  
 "config" 
  
 } 
  
 }, 
  
 { 
  
 "get_time_in_30s" 
 : 
  
 { 
  
 "assign" 
 : 
  
 [ 
  
 { 
  
 "now_plus_30s" 
 : 
  
 "${time.format(sys.now() + 30)}" 
  
 } 
  
 ] 
  
 } 
  
 }, 
  
 { 
  
 "start_run" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.startManualRuns" 
 , 
  
 "args" 
 : 
  
 { 
  
 "parent" 
 : 
  
 "${config.name}" 
 , 
  
 "body" 
 : 
  
 { 
  
 "requestedRunTime" 
 : 
  
 "${now_plus_30s}" 
  
 } 
  
 }, 
  
 "result" 
 : 
  
 "runsResp" 
  
 } 
  
 }, 
  
 { 
  
 "remove_run_config" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.delete" 
 , 
  
 "args" 
 : 
  
 { 
  
 "name" 
 : 
  
 "${config.name}" 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "delete_table" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.bigquery.v2.tables.delete" 
 , 
  
 "args" 
 : 
  
 { 
  
 "datasetId" 
 : 
  
 "${destination_dataset}" 
 , 
  
 "projectId" 
 : 
  
 "${project_id}" 
 , 
  
 "tableId" 
 : 
  
 "${destination_table}" 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "delete_dataset" 
 : 
  
 { 
  
 "call" 
 : 
  
 "googleapis.bigquery.v2.datasets.delete" 
 , 
  
 "args" 
 : 
  
 { 
  
 "projectId" 
 : 
  
 "${project_id}" 
 , 
  
 "datasetId" 
 : 
  
 "${destination_dataset}" 
  
 } 
  
 } 
  
 }, 
  
 { 
  
 "the_end" 
 : 
  
 { 
  
 "return" 
 : 
  
 "SUCCESS" 
  
 } 
  
 } 
  
 ] 
  
 } 
 }

Module: googleapis.bigquerydatatransfer.v1.projects.dataSources

Functions
`checkValidCreds`	Returns true if valid credentials exist for the given data source and requesting user.
`get`	Retrieves a supported data source and returns its settings.
`list`	Lists supported data sources and returns their settings.

Module: googleapis.bigquerydatatransfer.v1.projects.locations

Functions
`get`	Gets information about a location.
`list`	Lists information about the supported locations for this service.

Module: googleapis.bigquerydatatransfer.v1.projects.locations.dataSources

Functions
`checkValidCreds`	Returns true if valid credentials exist for the given data source and requesting user.
`get`	Retrieves a supported data source and returns its settings.
`list`	Lists supported data sources and returns their settings.

Module: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs

Functions
`create`	Creates a new data transfer configuration.
`delete`	Deletes a data transfer configuration, including any associated transfer runs and logs.
`get`	Returns information about a data transfer config.
`list`	Returns information about all transfer configs owned by a project in the specified location.
`patch`	Updates a data transfer configuration. All fields must be set, even if they are not updated.
`startManualRuns`	Start manual transfer runs to be executed now with schedule_time equal to current time. The transfer runs can be created for a time range where the run_time is between start_time (inclusive) and end_time (exclusive), or for a specific run_time.

Module: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.runs

Functions
`delete`	Deletes the specified transfer run.
`get`	Returns information about the particular transfer run.
`list`	Returns information about running and completed transfer runs.

Module: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.runs.transferLogs

Functions
`list`	Returns log messages for the transfer run.

Module: googleapis.bigquerydatatransfer.v1.projects.transferConfigs

Functions
`create`	Creates a new data transfer configuration.
`delete`	Deletes a data transfer configuration, including any associated transfer runs and logs.
`get`	Returns information about a data transfer config.
`list`	Returns information about all transfer configs owned by a project in the specified location.
`patch`	Updates a data transfer configuration. All fields must be set, even if they are not updated.
`startManualRuns`	Start manual transfer runs to be executed now with schedule_time equal to current time. The transfer runs can be created for a time range where the run_time is between start_time (inclusive) and end_time (exclusive), or for a specific run_time.

Module: googleapis.bigquerydatatransfer.v1.projects.transferConfigs.runs

Functions
`delete`	Deletes the specified transfer run.
`get`	Returns information about the particular transfer run.
`list`	Returns information about running and completed transfer runs.

Module: googleapis.bigquerydatatransfer.v1.projects.transferConfigs.runs.transferLogs

Functions
`list`	Returns log messages for the transfer run.

BigQuery Data Transfer API Connector Overview Stay organized with collections Save and categorize content based on your preferences.