Run a local Airflow environment with Composer Local Development CLI tool

Cloud Composer 3  |  Cloud Composer 2 |  Cloud Composer 1

This section describes how to create, configure, and run a local Airflow environment using Composer Local Development CLI tool.

About Composer Local Development CLI tool

Composer Local Development CLI tool streamlines Apache Airflow DAG development for Cloud Composer by running an Airflow environment locally. This local Airflow environment uses a Cloud Composer image used by a specific Cloud Composer version .

You can create a local Airflow environment based on an existing Cloud Composer environment. In this case, the local Airflow environment takes the list of installed PyPI packages and environment variable names from your Cloud Composer environment.

You can use this local Airflow environment for testing and development purposes, such as to test new DAG code, PyPI packages, or Airflow configuration options.

Before you begin

• Composer Local Development CLI tool creates local Airflow environments in a directory where you run the composer-dev create command. To access your local Airflow environment later, run the tool commands in the path where you initially created the local environment. All data for the local environment is stored in a subdirectory at the path where you created the local environment: ./composer/<local_environment_name> .

• Your computer must have enough disk space to store Cloud Composer images. Composer Local Development CLI tool stores one image file for each Cloud Composer version. For example, if you have two local Airflow environments with different Cloud Composer versions, then Composer Local Development CLI tool stores two Cloud Composer images.

• Composer Local Development CLI tool uses colorized output. You can disable colorized output with the NO_COLOR=1 variable: NO_COLOR=1 composer-dev <other commands> .

• If you have only one local environment, you can omit the local environment's name from all composer-dev commands, except the run-airflow-cmd .

• Install Composer Local Development CLI tool dependencies:

  • Python versions from 3.8 to 3.11 with pip
  • Google Cloud CLI

• Install Docker . Docker must be installed and running in the local system. To verify that Docker is running, you can run any Docker CLI command, such as docker ps .

Configure credentials

If not already done, get new user credentials to use for Application Default Credentials :

 gcloud  
auth  
application-default  
login 

Login in gcloud CLI using your Google account:

 gcloud  
auth  
login 

All API calls done by the Composer Local Development CLI tool and DAGs are executed from the account that you use in gcloud CLI. For example, if a DAG in your local Airflow environment reads contents of a Cloud Storage bucket, then this account must have permissions to access the bucket. This is different from Cloud Composer environments, where an environment's service account makes the calls.

Install Composer Local Development CLI tool

Clone the Composer Local Development CLI repository:

 git  
clone  
https://github.com/GoogleCloudPlatform/composer-local-dev.git 

In the top-level directory of the cloned repository, run:

 pip  
install  
. 

Depending on your pip configuration, the path at which the tool is installed might not be in PATH variable. If this is the case, pip displays a warning message. You can use information from this warning message to add this directory to the PATH variable in your operating system.

Create a local Airflow environment with a Cloud Composer image

To list available Cloud Composer images, run:

 composer-dev  
list-available-versions  
--include-past-releases  
--limit  
 10 
 

To create a local Airflow environment with default parameters, run:

 composer-dev  
create  
 \ 
  
--from-image-version  
 IMAGE_VERSION 
  
 \ 
  
 LOCAL_ENVIRONMENT_NAME 
 

Other parameters:

 composer-dev  
create  
 \ 
  
--from-image-version  
 IMAGE_VERSION 
  
 \ 
  
--project  
 PROJECT_ID 
  
 \ 
  
--port  
 WEB_SERVER_PORT 
  
 \ 
  
--dags-path  
 LOCAL_DAGS_PATH 
  
 \ 
  
 LOCAL_ENVIRONMENT_NAME 
 

Replace:

• IMAGE_VERSION with the name of the Cloud Composer image .
• PROJECT_ID with the Project ID .
• WEB_SERVER_PORT with the port that Airflow web server must listen at.
• LOCAL_DAGS_PATH with the path to a local directory where the DAG files are located.
• LOCAL_ENVIRONMENT_NAME with the name of this local Airflow environment.

Example:

 composer-dev  
create  
 \ 
  
--from-image-version  
composer-2.14.1-airflow-2.10.5  
 \ 
  
example-local-environment 

Create a local Airflow environment from a Cloud Composer environment

Only the following information is taken from a Cloud Composer environment:

• Versions of Cloud Composer and Airflow used in your environment.

• List of custom PyPI packages installed in your environment.

• Commented list of names of environment variables set in your environment.

Other information and configuration parameters from the environment, such as DAG files, DAG run history, Airflow variables, and connections, are not copied from your Cloud Composer environment.

To create a local Airflow environment from an existing Cloud Composer environment:

 composer-dev  
create  
 LOCAL_ENVIRONMENT_NAME 
  
 \ 
  
--from-source-environment  
 ENVIRONMENT_NAME 
  
 \ 
  
--location  
 LOCATION 
  
 \ 
  
--project  
 PROJECT_ID 
  
 \ 
  
--port  
 WEB_SERVER_PORT 
  
 \ 
  
--dags-path  
 LOCAL_DAGS_PATH 
 

Replace:

• LOCAL_ENVIRONMENT_NAME with a name for the local Airflow environment.
• ENVIRONMENT_NAME with the name of the Cloud Composer environment.
• LOCATION with the region where the Cloud Composer environment is located.
• PROJECT_ID with the Project ID .
• WEB_SERVER_PORT with a port for the local Airflow web server.
• LOCAL_DAGS_PATH with a path to a local directory where the DAGs are located.

Example:

 composer-dev  
create  
example-local-environment  
 \ 
  
--from-source-environment  
example-environment  
 \ 
  
--location  
us-central1  
 \ 
  
--project  
example-project  
 \ 
  
--port  
 8081 
  
 \ 
  
--dags-path  
example_directory/dags 

Start a local Airflow environment

To start a local Airflow environment, run:

 composer-dev  
start  
 LOCAL_ENVIRONMENT_NAME 
 

Replace:

• LOCAL_ENVIRONMENT_NAME with the name of a local Airflow environment.

Stop or restart a local Airflow environment

When you restart a local Airflow environment, Composer Local Development CLI tool restarts the Docker container where the environment runs. All Airflow components are stopped and started again. As a result, all DAG runs that are executed during a restart are marked as failed .

To restart or start a stopped local Airflow environment, run:

 composer-dev  
restart  
 LOCAL_ENVIRONMENT_NAME 
 

Replace:

• LOCAL_ENVIRONMENT_NAME with the name of a local Airflow environment.

To stop a local Airflow environment, run:

 composer-dev  
stop  
 LOCAL_ENVIRONMENT_NAME 
 

Add and update DAGs

Dags are stored in the directory that you specified in the --dags-path parameter when you created your local Airflow environment. By default, this directory is ./composer/<local_environment_name>/dags . You can get the directory used by your environment with the describe command .

To add and update DAGs, change files in this directory. You do not need to restart your local Airflow environment.

View local Airflow environment logs

You can view recent logs from a Docker container that runs your local Airflow environment. In this way, you can monitor container-related events and check Airflow logs for errors such as dependency conflicts caused by PyPI packages installation.

To view logs from a Docker container that runs your local Airflow environment, run:

 composer-dev  
logs  
 LOCAL_ENVIRONMENT_NAME 
  
--max-lines  
 10 
 

To follow the log stream, omit the --max-lines argument:

 composer-dev  
logs  
 LOCAL_ENVIRONMENT_NAME 
 

Run an Airflow CLI command

You can run Airflow CLI commands in your local Airflow environment.

To run an Airflow CLI command:

 composer-dev  
run-airflow-cmd  
 LOCAL_ENVIRONMENT_NAME 
  
 \ 
  
 SUBCOMMAND 
  
 SUBCOMMAND_ARGUMENTS 
 

Example:

 composer-dev  
run-airflow-cmd  
example-local-environment  
dags  
list  
-o  
table 

Configure local Airflow environments

Composer Local Development CLI tool stores configuration parameters for a local Airflow environment, such as environment variables and PyPI package requirements in the local environment's directory ( ./composer/<local_environment_name> ).

Configuration is applied when a local Airflow environment is started. For example, if you add conflicting PyPI package requirements, then Composer Local Development CLI tool reports errors when you start the local environment.

Airflow connections are stored in the database of the local Airflow environment. You can configure them by running an Airflow CLI command or by storing the connection parameters in environment variables . For more information about ways to create and configure connections, see Managing connections in the Airflow documentation.

Get a list and status of local Airflow environments

To list all available local Airflow environments and display their status:

 composer-dev  
list 

To describe a specific environment, and get details such as image version, DAGs path, and web server URL of an environment:

 composer-dev  
describe  
 LOCAL_ENVIRONMENT_NAME 
 

Replace:

• LOCAL_ENVIRONMENT_NAME with the name of the local Airflow environment.

List images used by local Airflow environments

To list all images used by Composer Local Development CLI tool, run:

 docker  
images  
--filter = 
 reference 
 = 
 '*/cloud-airflow-releaser/*/*' 
 

Install plugins and change data

Plugins and data for a local Airflow environment are taken from the local environment's directory: ./composer/<local_environment_name>/data and ./composer/<local_environment_name>/plugins ).

To change the contents of /data and /plugins directories, add or remove files in these directories. Docker automatically propagates file changes to your local Airflow environment.

Composer Local Development CLI tool does not support specifying a different directory for data and plugins.

Configure environment variables

To configure environment variables , edit the variables.env file in the environment directory: ./composer/<local_environment_name>/variables.env .

The variables.env file must contain key-value definitions, one line for each environment variable. To change Airflow configuration options, use the AIRFLOW__SECTION__KEY format. For more information about the available environment variables, see Airflow configuration reference .

  EXAMPLE_VARIABLE 
 = 
True ANOTHER_VARIABLE 
 = 
 test 
 AIRFLOW__WEBSERVER__DAG_DEFAULT_VIEW 
 = 
graph 

To apply the changes, restart your local Airflow environment .

Install or remove PyPI packages

To install or remove PyPI packages, modify the requirements.txt file in the environment directory: ./composer/<local_environment_name>/requirements.txt .

Requirements must follow the format specified in PEP-508 where each requirement is specified in lowercase and consists of the package name with optional extras and version specifiers.

To apply the changes, restart your local Airflow environment .

Switch to a different Cloud Composer image

You can use any Cloud Composer image with Composer Local Development CLI tool and switch between the images. This approach is different from upgrading your Cloud Composer environment, because configuration parameters of your local Airflow environment are applied when it starts.

For example, after a new Cloud Composer version is released, you can switch your environment to use it, and keep existing local Airflow environment configuration. As another example, you can switch between different Airflow versions within a specific Cloud Composer version.

To change the environment's image used by your local Airflow environment:

1. Edit the local environment configuration file: ./composer/<local_environment_name>/config.json .

2. Change the value of the composer_image_version parameter. To view available values, you can list available images .

3. To apply the changes, restart your local Airflow environment .

Delete a local Airflow environment

Caution:Make sure that you saved all required data from the environment, such as logs and configuration.

To delete a a local Airflow environment, run the following command:

 composer-dev  
remove  
 LOCAL_ENVIRONMENT_NAME 
 

If the environment is running, add the --force flag to force its removal.

Delete Docker images

To delete all images downloaded by Composer Local Development CLI tool, run:

 docker  
rmi  
 $( 
docker  
images  
--filter = 
 reference 
 = 
 '*/cloud-airflow-releaser/*/*' 
  
-q ) 
 

Troubleshooting

This section provides solutions to common issues.

Unable to start a local environment on MacOS

If you installed the composer-dev package to a directory where Docker cannot access it, then your local environment might not start.

For example, if Python is installed in the /opt directory, such as when you install it with default Homebrew configuration on MacOS, then the composer-dev package is also installed in the /opt directory. Because Docker complies with Apple's sandbox rules, the /opt directory isn't available by default. In addition, you cannot add it through the UI (Settings > Resources > File sharing).

In this case, Composer Local Development CLI tool generates an error message that is similar to the following example:

 Failed to create container with an error: 400 Client Error for ...
Bad Request ("invalid mount config for type "bind": bind source path does not exist:
/opt/homebrew/lib/python3.9/site-packages/composer_local_dev/docker_files/entrypoint.sh

Possible reason is that composer-dev was installed in the path that is
not available to Docker. See...") 

You can use one of the following solutions:

• Install Python or the composer-dev package to a different directory, so that Docker can access the package.
• Manually edit the ~/Library/Group\ Containers/group.com.docker/settings.json file and add /opt to filesharingDirectories .

What's next

• Write DAGs
• Test, synchronize, and deploy your DAGs from GitHub