Set up highly resilient environments

Cloud Composer 3  |  Cloud Composer 2 |  Cloud Composer 1

This page describes how to set up highly resilient (Highly Available) Cloud Composer environments.

About highly resilient (Highly Available) environments

A highly resilient environment is multi-zonal and runs across at least two zones of a selected region. Cloud Composer automatically distributes the components of your environment between zones.

For example, a zonal outage interrupts Airflow tasks that run in a specific zone. Afterwards, a highly resilient environment recovers, restarts its affected components in a different zone, and switches its database to a secondary zone. Thus, the failed Airflow tasks can be rescheduled and restarted by Airflow, while at the same time preserving the history of DAG runs and other settings.

About highly available database of your environment

In highly available Cloud Composer environments, the Cloud SQL component that stores the database of your environment runs in the high availability mode . A primary instance and a standby instance of Cloud SQL are distributed between zones.

In case of an outage, the Cloud SQL instance of your environment performs the automatic database failover to the standby Cloud SQL instance. You do not need to perform any additional actions in your Cloud Composer environment. Once the primary zone is operational again, the environment switches back to having two zones (primary and secondary). Primary and secondary zones can be swapped in some cases. The Cloud SQL instance in high availability mode uses the same IP address after a failover.

About highly available Airflow components

A highly resilient environment is multi-zonal and runs across at least two zones of a selected region. The following components run in separate zones:

• Exactly two Airflow schedulers

• At least two triggerers ( if the number of triggerers isn't set to zero )

• Two web servers

The minimum number of workers is set to two, and your environment's cluster distributes worker instances between zones. In case of a zonal outage, affected worker instances are rescheduled in a different zone.

For more information about the architecture of highly resilient environments, see Highly resilient environment architecture .

Before you begin

• In Cloud Composer 2, high resilience is available only in Private IP environments.

• Highly resilient environments are offered at an incremental charge when compared to regular environments.

• Highly resilient environments are available in Cloud Composer version 2.2.0 and later versions.

• If you want to update a standard environment to a highly resilient one, make sure that it meets the following configuration requirements. If your environment doesn't meet these requirements, you can update its scale and performance parameters .

  • The minimum number of Airflow workers is two or more.

  • The number of Airflow schedulers is exactly two.

  • If you use deferrable operators in your DAGs , then at least two triggerers.

Create a highly resilient environment

To create a highly resilient environment, enable the high resilience mode when you create an environment.

Update a standard environment to high resilience mode

   
gcloud  
composer  
environments  
update  
 ENVIRONMENT_NAME 
  
 \ 
  
--location  
 LOCATION 
  
 \ 
  
--enable-high-resilience 

  // PATCH https://composer.googleapis.com/v1/projects/example-project/ 
 // locations/us-central1/environments/example-environment?updateMask= 
 // config.resilienceMode 
 { 
  
 "config" 
 : 
  
 { 
  
 "resilience_mode" 
 : 
  
 { 
  
 "HIGH_RESILIENCE" 
  
 } 
  
 } 
  
 } 
 

  resource 
  
 "google_composer_environment" 
  
 "example" 
  
 { 
  
 provider 
  
 = 
  
 google-beta 
  
 name 
  
 = 
  
 " ENVIRONMENT_NAME 
" 
  
 region 
  
 = 
  
 " LOCATION 
" 
  
 config 
  
 { 
  
 resilience_mode 
  
 = 
  
 "HIGH_RESILIENCE" 
  
 } 
 } 
 

  resource 
  
 "google_composer_environment" 
  
 "example" 
  
 { 
  
 provider 
  
 = 
  
 google-beta 
  
 name 
  
 = 
  
 "example-environment" 
  
 region 
  
 = 
  
 "us-central1" 
  
 config 
  
 { 
  
 resilience_mode 
  
 = 
  
 "HIGH_RESILIENCE" 
 } 
 

Change a highly resilient environment to standard resilience mode

You can change your environment to standard resilience mode at any time. This operation:

• Reduces the number of web servers in your environment to one.
• Switches off the high availability mode of your environment's Airflow database.

• Doesn't change the settings for minimum number of Airflow workers, schedulers, or triggerers.

   
gcloud  
composer  
environments  
update  
 ENVIRONMENT_NAME 
  
 \ 
  
--location  
 LOCATION 
  
 \ 
  
--disable-high-resilience 

  // PATCH https://composer.googleapis.com/v1/projects/example-project/ 
 // locations/us-central1/environments/example-environment?updateMask= 
 // config.resilienceMode 
 { 
  
 "config" 
 : 
  
 { 
  
 "resilience_mode" 
 : 
  
 { 
  
 "RESILIENCE_MODE_UNSPECIFIED" 
  
 } 
  
 } 
  
 } 
 

  resource 
  
 "google_composer_environment" 
  
 "example" 
  
 { 
  
 provider 
  
 = 
  
 google-beta 
  
 name 
  
 = 
  
 " ENVIRONMENT_NAME 
" 
  
 region 
  
 = 
  
 " LOCATION 
" 
  
 config 
  
 { 
  
 resilience_mode 
  
 = 
  
 "STANDARD_RESILIENCE" 
  
 } 
 } 
 

  resource 
  
 "google_composer_environment" 
  
 "example" 
  
 { 
  
 provider 
  
 = 
  
 google-beta 
  
 name 
  
 = 
  
 "example-environment" 
  
 region 
  
 = 
  
 "us-central1" 
  
 config 
  
 { 
  
 resilience_mode 
  
 = 
  
 "STANDARD_RESILIENCE" 
 } 
 

Check if your environment runs in the high resilience mode

 gcloud  
composer  
environments  
describe  
 ENVIRONMENT_NAME 
  
 \ 
  
--location  
 LOCATION 
  
 \ 
  
--format = 
 "value(config.resilienceMode)" 
 

What's next

• Perform failover tests for your highly resilient environment.
• Disaster recovery with environment snapshots