Deployment configuration samples

The config/config.yaml file — typically initialized from the config/config.yaml.example template — serves as the primary configuration for the Cortex Framework deployment. It defines critical parameters including the target Google Cloud execution project, source and destination BigQuery datasets, and Dataform specifications such as repository and workspace names.

The following sections provide a detailed breakdown of the config/config.yaml structure.

Build environment

The build environment project is the project that gets billed for build actions, such as BigQuery jobs (reading DD03L ).

  buildEnvironment 
 : 
  
 buildProjectId 
 : 
  
  YOUR_BUILD_PROJECT_ID 
 
 

The following table describes the build environment parameters.

Data

The data: section of the configuration file defines your data sources, targets, and the specific modules for the data foundation and data products. Its general structure is as follows:

  data 
 : 
  
 # Geographic location for BigQuery datasets (for example: US, EU, us-central1) 
  
 # For full list see: https://docs.cloud.google.com/cortex/docs/supported-locations 
  
 bigQueryLocation 
 : 
  
 US 
  
 # List of namespaces for data foundation and product modules. 
  
 namespaces 
 : 
  
 - 
  
 name 
 : 
  
 cortex 
  
 path 
 : 
  
 cortex 
  
 # List of source datasets. 
  
 sources 
 : 
  
 - 
  
 ... 
  
 # List of target datasets. 
  
 targets 
 : 
  
 - 
  
 ... 
  
 # Configuration for data foundation and product modules. 
  
 modules 
 : 
  
 # List of foundation modules. 
  
 foundation 
 : 
  
 - 
  
 ... 
  
  
 # List of data product modules. 
  
 product 
 : 
  
 - 
  
 ... 
 

Data: BigQuery location

Defines the location of the BigQuery source and target datasets.

Data: Cortex namespace

Defines Cortex Framework namespace.

Data: BigQuery sources and target datasets

The list of sources defines BigQuery datasets where the raw data from the source system has been replicated or streamed into.

The targets define a list of BigQuery datasets where the Dataform processed datasets will be stored.

Each of source and targets are referenced from the modules using its unique ID.

  # Data source and target mapping 
 sources 
 : 
  
 - 
  
 id 
 : 
  
 sap_raw 
  
 projectId 
 : 
  
  YOUR_SOURCE_PROJECT_ID 
 
  
 datasetId 
 : 
  
 cortex_sap_raw 
 targets 
 : 
  
 - 
  
 id 
 : 
  
 sap_foundation 
  
 projectId 
 : 
  
  YOUR_TARGET_PROJECT_ID 
 
  
 datasetId 
 : 
  
 cortex7_sap_data_foundation 
 

The following table describes the data source and target mapping parameters.

Data: Modules

The modules define the structure and components of the Dataform data pipelines.

Data: Modules: Foundation

This section configures the data foundation layer modules that process data from the raw layer (CDC streams) into standardized latest records representation of the source data. In case the source provides a view on the latest records directly, or such transformations are performed by the source system connector, the module can be configured as an external data foundation source.

  modules 
 : 
  
 # List of foundation modules. 
  
 foundation 
 : 
  
 # Unique identifier for the module instance. 
  
 - 
  
 moduleId 
 : 
  
 erp 
  
 # Type of the module (namespaced, for example, cortex.sap). 
  
 type 
 : 
  
 cortex.sap 
  
 # Reference to the source dataset ID. 
  
 dataSourceId 
 : 
  
 sap_raw 
  
 # Reference to the target dataset ID. 
  
 dataTargetId 
 : 
  
 sap_foundation 
  
 # Module-specific configuration settings. 
  
 moduleSettings 
 : 
  
 # SAP version (for example, ecc, s4). 
  
 sapVersion 
 : 
  
 ecc 
  
 # SAP client number. 
  
 mandt 
 : 
  
 "100" 
  
 # Whether the module is enabled. 
  
 # enabled: true 
  
 # Whether the foundation is external (does not create target dataset). 
  
 # external: false 
  
 # Path to the table settings configuration file. 
  
 # tableSettings: "config/data_foundation/sap/table_settings.yaml" 
 

The following table describes the data foundation modules parameters for modules.foundation configuration.

Data: Modules: Data products

Data product modules define the aggregations, calculations, and joins necessary to transform raw data into insights that fulfill specific business use cases.

The configuration of the data products allows setting of unique ID, definition of dependencies as well as reference of the data foundation module and target dataset where the results will be stored into.

Detailed configuration of given data products is defined within files referenced by the key: tableSettings .

  modules 
 : 
  
 # List of data product modules. 
  
 product 
 : 
  
 # Unique identifier for the data product instance. 
  
 - 
  
 moduleId 
 : 
  
 sap_purchasing_organizations 
  
 # Type of the data product (namespaced). 
  
 type 
 : 
  
 cortex.purchasing_organizations 
  
 # Map of module dependencies. 
  
 dependsOn 
 : 
  
 sapModule 
 : 
  
 erp 
  
 # Reference to the target dataset ID. 
  
 dataTargetId 
 : 
  
 product_target 
  
 # Whether the module is enabled. 
  
 # enabled: true 
  
 # Path to the table settings configuration file. 
  
 # tableSettings:   "config/cortex/data_product/purchasing_organizations/table_settings.yaml" 
 

The following table describes the data product modules parameters for modules.product configuration.

Deployment environment

Cortex Framework uses Dataform to orchestrate SQL transformations within BigQuery. The deployment: block defines the Dataform configuration, responsible for the execution of the data pipelines, including the repository project, location, repository name, and the Dataform workspace name.

  deployment 
 : 
  
 targets 
 : 
  
 - 
  
 type 
 : 
  
 dataform 
  
 enabled 
 : 
  
 true 
  
 targetSettings 
 : 
  
 repositoryProjectId 
 : 
  
  YOUR_REPO_PROJECT_ID 
 
  
 repositoryRegion 
 : 
  
 us-central1 
  
 repositoryName 
 : 
  
 cortex-repository 
  
 workspaceName 
 : 
  
 dev 
 

The following table describes the deployment targets location parameters ( deployment.targets: ).