Create data integration workflows using the Pipelines & Connections page

The BigQuery Pipelines & Connectionspage in the console streamlines your data integration tasks by providing guided, BigQuery-specific configuration workflows for services like BigQuery Data Transfer Service, Datastream, and Pub/Sub.

You can use the Pipelines & Connectionspage to perform the following tasks:

• Loading data. Find and load data from supported data sources using a guided experience.
• Scheduling data tasks. Schedule your data processes to run automatically, including scheduled queries, notebooks, data preparations, and pipelines.
• Transforming data. Create pipelines and data preparations that perform operations such as cleaning, structuring, and readying your data for analysis and reporting.
• Creating connections to data sources. Connect to external data sources, compute runtimes, and AI services for data access and remote execution.

After you create your tasks, the Pipelines & Connectionspage provides a single location for managing and monitoring your assets. You can see the status of your assets in the list view, and you can use the detailed monitoring pages to see operational metrics for each configured asset such as scheduled queries, pipelines, transfers, connections, streams, and subscriptions.

Operational metrics include BigQuery Data Transfer Service run histories, throughput and latency for Pub/Sub subscriptions, and data freshness and event counts for Datastream pipelines.

The monitoring pages also include troubleshooting information such as logs for BigQuery Data Transfer Service runs and Datastream object status.

Opt in or out

During Preview, you can opt into or out of using the Pipelines & Connectionspage.

Opt in

To opt into using the Pipelines & Connectionspage, follow these steps:

1. Go to the Google Cloud console.

   Go to the console

2. In the Google Cloud console toolbar, click menu Navigation menu.

3. Click Solutions > All products.

4. In the Analyticssection, click BigQuery.

   The BigQuery Studiopage opens.

5. To expand the navigation menu, click last_page Toggle BigQuery navigation menu.

   

6. In the navigation menu, click one of the following options:

   • Data transfers
   • Scheduled queries
   • Scheduling

7. Go to the unified pipelines and connections banner.

   

8. Click Opt-in.

   After you opt in, the existing Data transfers, Scheduling, and Scheduled queriespages are replaced by the Pipelines & Connectionspage.

Opt out

To opt out of using the unified Pipelines & Connectionspage, follow these steps:

1. On the Pipelines & Connectionspage, click Opt-outin the unified pipelines and connections banner.

2. A feedback dialog opens. Provide feedback on why you're opting out such as missing features or bugs.

3. Click Send, and then click Close.

   After you submit the form, your navigation menu reverts back to using separate Data transfers, Scheduling, and Scheduled queriespages.

Create a data integration asset

To create assets such as scheduled queries, data transfers, and external connections, do the following:

1. Go to the Pipelines & Connectionspage.

   Go to Pipelines & Connections

2. Click Create.

3. For the asset you want to create, click the option in the card:

   1. To add data from one of the supported data sources using a guided workflow, in the Load datacard, click Add data.

   2. To schedule a query, notebook, data preparation, or pipeline, in the Schedule data taskscard, click Schedule, and then choose the appropriate option.

   3. To create a transformation pipeline or data preparation, in the Transform datacard, click Create, and then choose the appropriate option.

   4. To create a connection to an external data source, in the External connectionscard, click Create connection.

Monitor your assets

After you create your assets, you can view and monitor them.

1. Go to the Pipelines & Connectionspage.

   Go to Pipelines & Connections

2. To view details about your connections, click the Connectionstab.

3. To monitor your pipelines, click the Pipelinestab. You can use this page to view details such as the pipeline's status, type, and run history.

4. To view a pipeline's details, in the Display namecolumn, click the link.

What's next

• Learn how to create a Dataform pipeline in BigQuery .
• Learn how to schedule a pipeline .
• Learn how to schedule a query .
• Learn about BigQuery Data Transfer Service transfers .