Migrate environments to Cloud Composer 2 (from Airflow 2) using snapshots

Cloud Composer 3  |  Cloud Composer 2  |  Cloud Composer 1

This page explains how to transfer DAGs, data and configuration from your existing Cloud Composer 1, Airflow 2 environments to Cloud Composer 2, Airflow 2.

This migration guide uses the Snapshots feature.

Other migration guides

Before you begin

• Snapshots are supported in Cloud Composer 2 version 2.0.9 and later. Cloud Composer 1 supports saving environment snapshots in 1.18.5.

• Cloud Composer supports side-by-side migration from Cloud Composer 1 to Cloud Composer 2. It is not possible to upgrade from Cloud Composer 1 to Cloud Composer 2 in-place.

• Check the list of differences between Cloud Composer 1 and Cloud Composer 2 .

• The maximum size of the Airflow database that supports snapshots is 20 GB. If your environment's database takes more than 20 GB, reduce the size of the Airflow database .

• The total number of objects in the /dags , /plugins and /data folders in the environment's bucket must be less than 100,000 to create snapshots.

• If you use the XCom mechanism to transfer files, make sure that you use it according to Airflow's guidelines . Transferring big files or a large number of files using XCom impacts Airflow database's performance and can lead to failures when loading snapshots or upgrading your environment. Consider using alternatives such as Cloud Storage to transfer large volumes of data.

Step 1: Pause DAGs in your Cloud Composer 1 environment

To avoid duplicate DAG runs, pause all DAGs in your Cloud Composer 1 environment before saving its snapshot.

You can use any of the following options:

• In the Airflow web interface , go to DAGsand pause all DAGs manually.

• Use the composer_dags script to pause all DAGs:

   python3  
  composer_dags.py  
  --environment  
   COMPOSER_1_ENV 
    
   \ 
    
  --project  
   PROJECT_ID 
    
   \ 
    
  --location  
   COMPOSER_1_LOCATION 
    
   \ 
    
  --operation  
  pause 
  

  Replace:

  • COMPOSER_1_ENV with the name of your Cloud Composer 1 environment.
  • PROJECT_ID with the Project ID .
  • COMPOSER_1_LOCATION with the region where the environment is located.

• (Airflow versions 2.9.1 and later) If there are quota errors while pausing a large number of DAGs, you can use the following Airflow CLI commands to pause all DAGs at once:

   gcloud  
  composer  
  environments  
  run  
   COMPOSER_1_ENV 
    
  dags  
  pause  
   \ 
    
  --project  
   PROJECT_ID 
    
   \ 
    
  --location  
   COMPOSER_1_LOCATION 
    
   \ 
    
  --  
  -y  
  --treat-dag-id-as-regex  
   ".*" 
   
  

• (Airflow versions earlier than 2.9.1) If there are quota errors while pausing a large number of DAGs, it's possible to pause DAGs using the Airflow REST API . Also see Trying the API in the Airflow documentation.

Step 2: Save the snapshot of your Cloud Composer 1 environment

Step 3: Create a Cloud Composer 2 environment

Create a Cloud Composer 2 environment . You can start with an environment preset that matches your expected resource demands, and later scale and optimize your environment further.

You do not need to specify configuration overrides and environment variables, since you replace them later when you load the snapshot of your Cloud Composer 1 environment.

Step 4: Load the snapshot to your Cloud Composer 2 environment

 gcloud  
composer  
environments  
snapshots  
load  
 \ 
  
 COMPOSER_2_ENV 
  
 \ 
  
--location  
 COMPOSER_2_LOCATION 
  
 \ 
  
--snapshot-path  
 " SNAPSHOT_PATH 
" 
 

Step 5: Unpause DAGs in the Cloud Composer 2 environment

You can use any of the following options:

• In the Airflow web interface , go to DAGsand unpause all DAGs manually one by one.

• Use the composer_dags script to unpause all DAGs:

   python3  
  composer_dags.py  
  --environment  
   COMPOSER_2_ENV 
    
   \ 
    
  --project  
   PROJECT_ID 
    
   \ 
    
  --location  
   COMPOSER_2_LOCATION 
    
   \ 
    
  --operation  
  unpause 
  

  Replace:

  • COMPOSER_2_ENV with the name of your Cloud Composer 2 environment.
  • PROJECT_ID with the Project ID .
  • COMPOSER_2_LOCATION with the region where the environment is located.

• (Airflow versions 2.9.1 and later) If there are quota errors while unpausing a large number of DAGs, you can use the following Airflow CLI commands to unpause all DAGs at once:

   gcloud  
  composer  
  environments  
  run  
   COMPOSER_2_ENV 
    
  dags  
  unpause  
   \ 
    
  --project  
   PROJECT_ID 
    
   \ 
    
  --location  
   COMPOSER_2_LOCATION 
    
   \ 
    
  --  
  -y  
  --treat-dag-id-as-regex  
   ".*" 
   
  

• (Airflow versions earlier than 2.9.1) If there are quota errors while unpausing a large number of DAGs, it's possible to unpause DAGs using the Airflow REST API . Also see Trying the API in the Airflow documentation.

Step 6: Check for DAG errors

1. In the Airflow web interface , go to DAGsand check for reported DAG syntax errors.

2. Check that DAG runs are scheduled at the correct time.

3. Wait for the DAG runs to happen in the Cloud Composer 2 environment and check if they were successful. If a DAG run was successful, do not unpause it in the Cloud Composer 1 environment; if you do so, a DAG run for the same time and date happens in your Cloud Composer 1 environment.

4. If a specific DAG runs fails, attempt to troubleshoot the DAG until it successfully runs in Cloud Composer 2.

Step 7: Monitor your Cloud Composer 2 environment

After you transfer all DAGs and configuration to the Cloud Composer 2 environment, monitor it for potential issues, failed DAG runs, and overall environment health.

If the Cloud Composer 2 environment runs without problems for a sufficient period of time, consider deleting the Cloud Composer 1 environment.

What's next

• Troubleshooting DAGs
• Troubleshooting environment creation
• Troubleshooting environment updates
• Using backport packages