Migrate environments to Managed Airflow (Gen 2) (from Airflow 1) using snapshots

Managed Airflow (Gen 3)  |  Managed Airflow (Gen 2)  |  Managed Airflow (Legacy Gen 1)

This page explains how to transfer DAGs, data and configuration from your existing Managed Airflow (Legacy Gen 1), Airflow 1 environments to Managed Airflow (Gen 2), Airflow 2.

This migration guide uses the Snapshots feature.

Other migration guides

Before you begin

• Snapshots are supported in Managed Airflow (Gen 2) version 2.0.9 and later. Managed Airflow (Legacy Gen 1) supports saving environment snapshots in 1.18.5.

• Managed Airflow supports side-by-side migration from Managed Airflow (Legacy Gen 1) to Managed Airflow (Gen 2). It is not possible to upgrade from Managed Airflow (Legacy Gen 1) to Managed Airflow (Gen 2) in-place.

• Check the list of differences between Managed Airflow (Legacy Gen 1) and Managed Airflow (Gen 2) .

• The maximum size of the Airflow database that supports snapshots is 20 GB. If your environment's database takes more than 20 GB, reduce the size of the Airflow database .

• The total number of objects in the /dags , /plugins and /data folders in the environment's bucket must be less than 100,000 to create snapshots.

• If you use the XCom mechanism to transfer files, make sure that you use it according to Airflow's guidelines . Transferring big files or a large number of files using XCom impacts Airflow database's performance and can lead to failures when loading snapshots or upgrading your environment. Consider using alternatives such as Cloud Storage to transfer large volumes of data.

• Because Managed Airflow (Gen 2) uses Airflow 2, the migration includes switching your DAGs and environment configuration to Airflow 2. Check the migration guide from Airflow 1 to Airflow 2 for information about the breaking changes between Airflow 1 and Airflow 2 in Managed Airflow.

• In this guide, you combine migration to Airflow 2 and migration to Managed Airflow (Gen 2) in one migration procedure. In this way, you do not need to migrate to a Managed Airflow (Legacy Gen 1) environment with Airflow 2 before migrating to Managed Airflow (Gen 2).

Step 1: Upgrade to Airflow 1.10.15

If your environment uses an Airflow version earlier than 1.10.15, upgrade your environment to a Managed Airflow version that uses Airflow 1.10.15 and supports snapshots.

Step 2: Check compatibility with Airflow 2

To check for potential conflicts with Airflow 2, consult the Upgrading to Airflow 2.0+ guide, in the section about upgrading DAGs .

One common issue that you might encounter is related to incompatible import paths. For more information about solving this compatibility issue, in the Upgrading to Airflow 2.0+ guide, see the section about backport providers .

Step 3: Make sure that your DAGs are ready for Airflow 2

Before transferring DAGs to your Managed Airflow (Gen 2) environment, make sure that:

1. Your DAGs run successfully and there are no remaining compatibility issues.

2. Your DAGs use correct import statements .

   For example, the new import statement for BigQueryCreateDataTransferOperator can look like this:

     from 
     
    airflow.providers.google.cloud.operators.bigquery_dts 
   \ import 
     
    BigQueryCreateDataTransferOperator 
    
   

3. Your DAGs are upgraded for Airflow 2 . This change is compatible with Airflow 1.10.14 and later versions.

Step 4: Pause DAGs in your Managed Airflow (Legacy Gen 1) environment

To avoid duplicate DAG runs, pause all DAGs in your Managed Airflow (Legacy Gen 1) environment before saving its snapshot.

You can use any of the following options:

• In the Airflow web interface , go to DAGsand pause all DAGs manually.

• Use the composer_dags script to pause all DAGs:

   python3  
  composer_dags.py  
  --environment  
   COMPOSER_1_ENV 
    
   \ 
    
  --project  
   PROJECT_ID 
    
   \ 
    
  --location  
   COMPOSER_1_LOCATION 
    
   \ 
    
  --operation  
  pause 
  

  Replace:

  • COMPOSER_1_ENV with the name of your Managed Airflow (Legacy Gen 1) environment.
  • PROJECT_ID with the Project ID .
  • COMPOSER_1_LOCATION with the region where the environment is located.

Step 5: Save the snapshot of your Managed Airflow (Legacy Gen 1) environment

Step 6: Create a Managed Airflow (Gen 2) environment

Create a Managed Airflow (Gen 2) environment . You can start with an environment preset that matches your expected resource demands, and later scale and optimize your environment further.

You do not need to specify configuration overrides and environment variables, since you replace them later when you load the snapshot of your Managed Airflow (Legacy Gen 1) environment.

Some configuration options from Airflow 1 use a different name and section in Airflow 2. For more information, see Configuration changes .

Step 7: Load the snapshot to your Managed Airflow (Gen 2) environment

 gcloud  
composer  
environments  
snapshots  
load  
 \ 
  
 COMPOSER_2_ENV 
  
 \ 
  
--location  
 COMPOSER_2_LOCATION 
  
 \ 
  
--snapshot-path  
 " SNAPSHOT_PATH 
" 
 

Step 8: Unpause DAGs in the Managed Airflow (Gen 2) environment

You can use any of the following options:

• In the Airflow web interface , go to DAGsand unpause all DAGs manually one by one.

• Use the composer_dags script to unpause all DAGs:

   python3  
  composer_dags.py  
  --environment  
   COMPOSER_2_ENV 
    
   \ 
    
  --project  
   PROJECT_ID 
    
   \ 
    
  --location  
   COMPOSER_2_LOCATION 
    
   \ 
    
  --operation  
  unpause 
  

  Replace:

  • COMPOSER_2_ENV with the name of your Managed Airflow (Gen 2) environment.
  • PROJECT_ID with the Project ID .
  • COMPOSER_2_LOCATION with the region where the environment is located.

• (Airflow versions 2.9.1 and later) If there are quota errors while unpausing a large number of DAGs, you can use the following Airflow CLI commands to unpause all DAGs at once:

   gcloud  
  composer  
  environments  
  run  
   COMPOSER_2_ENV 
    
  dags  
  unpause  
   \ 
    
  --project  
   PROJECT_ID 
    
   \ 
    
  --location  
   COMPOSER_2_LOCATION 
    
   \ 
    
  --  
  -y  
  --treat-dag-id-as-regex  
   ".*" 
   
  

• (Airflow versions earlier than 2.9.1) If there are quota errors while unpausing a large number of DAGs, it's possible to unpause DAGs using the Airflow REST API . Also see Trying the API in the Airflow documentation.

Step 9: Check for DAG errors

1. In the Airflow web interface , go to DAGsand check for reported DAG syntax errors.

2. Check that DAG runs are scheduled at the correct time.

3. Wait for the DAG runs to happen in the Managed Airflow (Gen 2) environment and check if they were successful. If a DAG run was successful, do not unpause it in the Managed Airflow (Legacy Gen 1) environment; if you do so, a DAG run for the same time and date happens in your Managed Airflow (Legacy Gen 1) environment.

4. If a specific DAG runs fails, attempt to troubleshoot the DAG until it successfully runs in Managed Airflow (Gen 2).

Step 10: Monitor your Managed Airflow (Gen 2) environment

After you transfer all DAGs and configuration to the Managed Airflow (Gen 2) environment, monitor it for potential issues, failed DAG runs, and overall environment health.

If the Managed Airflow (Gen 2) environment runs without problems for a sufficient period of time, consider deleting the Managed Airflow (Legacy Gen 1) environment.

What's next

• Troubleshooting DAGs
• Troubleshooting environment creation
• Troubleshooting environment updates
• Using backport packages