Introduction to BigQuery pipelines
You can use BigQuery pipelines to automate and streamline your BigQuery data processes. With pipelines, you can schedule and execute code assets in sequence to improve efficiency and reduce manual effort.
Overview
Pipelines are powered by Dataform .
A pipeline consists of one or more of the following code assets:
You can use pipelines to schedule the execution of code assets. For example, you can schedule a SQL query to run daily and update a table with the most recent source data, which can then power a dashboard.
In a pipeline with multiple code assets, you define the execution sequence. For example, to train a machine learning model, you can create a workflow in which a SQL query prepares data, and then a subsequent notebook trains the model using that data.
Capabilities
You can do the following in a pipeline:
- Create new or import existing SQL queries or notebooks into a pipeline.
- Schedule a pipeline to automatically run at a specified time and frequency.
- Share a pipeline with users or groups you specify.
- Share a link to a pipeline .
Limitations
Pipelines are subject to the following limitations:
- Pipelines are available only in the Google Cloud console.
- You can't change the region for storing a pipeline after it is created.
- You can grant users or groups access to a selected pipeline, but you can't grant them access to individual tasks within the pipeline.
Set the default region for code assets
If this is the first time you are creating a code asset, you should set the default region for code assets. You can't change the region for a code asset after it is created.
All code assets in BigQuery Studio use the same default region. To set the default region for code assets, follow these steps:
-
Go to the BigQuerypage.
-
In the Explorerpane, find the project in which you have enabled code assets.
-
Click View actionsnext to the project, and then click Change my default code region.
-
For Region, select the region that you want to use for code assets.
-
Click Select.
For a list of supported regions, see BigQuery Studio locations .
Supported regions
All code assets are stored in your default region for code assets . Updating the default region changes the region for all code assets created after that point.
The following table lists the regions where pipelines are available:
africa-south1
us-east5
us-west2
us-west4
us-east4
us-east1
asia-east2
asia-southeast2
asia-south1
asia-northeast3
asia-southeast1
australia-southeast1
asia-east1
asia-northeast1
europe-west3
europe-west12
me-central1
me-central2
Quotas and limits
BigQuery pipelines are subject to Dataform quotas and limits .
Pricing
The execution of BigQuery pipeline tasks incurs compute and storage charges in BigQuery. For more information, see BigQuery pricing .
Pipelines containing notebooks incur Colab Enterprise runtime charges based on the default machine type . For pricing details, see Colab Enterprise pricing .
Each BigQuery pipeline run is logged using Cloud Logging . Logging is automatically enabled for BigQuery pipeline runs, which can incur Cloud Logging billing charges. For more information, see Cloud Logging pricing .
What's next
- Learn how to create pipelines .
- Learn how to manage pipelines .
- Learn how to schedule pipelines .
