Stay organized with collectionsSave and categorize content based on your preferences.
Dataflow has ended support for Dataflow SQL.
This is a breaking change.
As of July 31, 2024, you can't access Dataflow SQL in the
Google Cloud console.
As of January 31, 2025, you can't use Dataflow SQL in the
Google Cloud CLI.
Replacement: Beam SQL
Beam SQLlets you
query bounded and unboundedPCollectionswith SQL statements. If you're
working in an Apache Beam notebook, you can use the IPythoncustom magicbeam_sqlto speed up your pipeline development.
Beam SQL provides similar functionality as Dataflow SQL,
using SQL statements to queryPCollections.
Migration
We recommend that you begin migrating your Dataflow SQL query to Beam SQL
as soon as possible.
Moving to Beam SQL is a breaking change that
affects any Dataflow SQL job or query.You will need to rebuild these
jobs and queries.
With this change, you can't create new Dataflow SQL jobs. However, this
change doesn't affect Dataflow SQL jobs that are already running.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eDataflow SQL is losing support, effective July 31, 2024, in the Google Cloud console and January 31, 2025, in the Google Cloud CLI.\u003c/p\u003e\n"],["\u003cp\u003eBeam SQL is the recommended replacement for Dataflow SQL, offering similar functionality for querying \u003ccode\u003ePCollections\u003c/code\u003e with SQL.\u003c/p\u003e\n"],["\u003cp\u003eMigrating from Dataflow SQL to Beam SQL requires rebuilding existing jobs and queries, as it is a breaking change.\u003c/p\u003e\n"],["\u003cp\u003eNew Dataflow SQL jobs cannot be created, but this change does not impact previously existing, currently running Dataflow SQL jobs.\u003c/p\u003e\n"]]],[],null,["# Dataflow SQL\n\nDataflow has ended support for Dataflow SQL. This is a breaking change.\n\n\u003cbr /\u003e\n\n- As of July 31, 2024, you can't access Dataflow SQL in the Google Cloud console.\n- As of January 31, 2025, you can't use Dataflow SQL in the Google Cloud CLI.\n\nReplacement: Beam SQL\n---------------------\n\n[Beam SQL](/dataflow/docs/guides/notebook-advanced#beam-sql) lets you\nquery bounded and unbounded `PCollections` with SQL statements. If you're\nworking in an Apache Beam notebook, you can use the IPython\n[custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html)\n`beam_sql` to speed up your pipeline development.\n\nBeam SQL provides similar functionality as Dataflow SQL,\nusing SQL statements to query `PCollections`.\n\nMigration\n---------\n\nWe recommend that you begin migrating your Dataflow SQL query to Beam SQL\nas soon as possible.\n\nMoving to Beam SQL is a breaking change that\naffects any Dataflow SQL job or query. **You will need to rebuild these\njobs and queries.**\n\nWith this change, you can't create new Dataflow SQL jobs. However, this\nchange doesn't affect Dataflow SQL jobs that are already running.\n\nFor information about creating queries in Beam SQL, see\n[Beam SQL and `beam_sql` magic](/dataflow/docs/guides/notebook-advanced#beam-sql)\nand\n[Beam SQL overview](https://beam.apache.org/documentation/dsls/sql/overview/)."]]
