Specifying machine types or scale tiers

When running a training job on AI Platform Training you must specify the number and types of machines you need. To make the process easier, you can pick from a set of predefined cluster specifications called scale tiers . Alternatively, you can choose a custom tier and specify the machine types yourself.

Specifying your configuration

How you specify your cluster configuration depends on how you plan to run your training job:

trainingInput:
  scaleTier: CUSTOM
  masterType: n1-highcpu-16
  workerType: n1-highcpu-16
  parameterServerType: n1-highmem-8
  evaluatorType: n1-highcpu-16
  workerCount: 9
  parameterServerCount: 3
  evaluatorCount: 1

gcloud ai-platform jobs submit training $JOB_NAME \
        --package-path $TRAINER_PACKAGE_PATH \
        --module-name $MAIN_TRAINER_MODULE \
        --job-dir $JOB_DIR \
        --region $REGION \ --config config.yaml \ 
-- \
        --user_first_arg=first_arg_value \
        --user_second_arg=second_arg_value

 gcloud ai-platform jobs submit training $JOB_NAME \
        --package-path $TRAINER_PACKAGE_PATH \
        --module-name $MAIN_TRAINER_MODULE \
        --job-dir $JOB_DIR \
        --region $REGION \
        --scale-tier custom \
        --master-machine-type n1-highcpu-16 \
        --worker-machine-type n1-highcpu-16 \
        --parameter-server-machine-type n1-highmem-8 \
        --worker-count 9 \
        --parameter-server-count 3 \
        -- \
        --user_first_arg=first_arg_value \
        --user_second_arg=second_arg_value 

training_inputs = {'scaleTier': 'CUSTOM',
    'masterType': 'n1-highcpu-16',
    'workerType': 'n1-highcpu-16',
    'parameterServerType': 'n1-highmem-8',
    'evaluatorType': 'n1-highcpu-16',
    'workerCount': 9,
    'parameterServerCount': 3,
    'evaluatorCount': 1,
    'packageUris': ['gs://my/trainer/path/package-0.0.0.tar.gz'],
    'pythonModule': 'trainer.task'
    'args': ['--arg1', 'value1', '--arg2', 'value2'],
    'region': 'us-central1',
    'jobDir': 'gs://my/training/job/directory',
    'runtimeVersion': '2.11',
    'pythonVersion': '3.7'}

job_spec = {'jobId': my_job_name, 'trainingInput': training_inputs}

Scale tiers

Google may optimize the configuration of the scale tiers for different jobs over time, based on customer feedback and the availability of cloud resources. Each scale tier is defined in terms of its suitability for certain types of jobs. Generally, the more advanced the tier, the more machines are allocated to the cluster, and the more powerful the specifications of each virtual machine. As you increase the complexity of the scale tier, the hourly cost of training jobs, measured in training units , also increases. See the pricing page to calculate the cost of your job.

AI Platform Training does not support distributed training or training with accelerators for scikit-learn or XGBoost code. If your training job runs scikit-learn or XGBoost code, you must set the scale tier to either BASIC or CUSTOM .

Below are the scale tier identifiers:

Machine types for the custom scale tier

Use a custom scale tier for finer control over the processing cluster that you use to train your model. Specify the configuration in the TrainingInput object in your job configuration. If you're using the gcloud ai-platform jobs submit training command to submit your training job, you can use the same identifiers:

• Set the scale tier ( scaleTier ) to CUSTOM .

• Set values for the number of workers ( workerCount ), parameter servers ( parameterServerCount ), and evaluators ( evaluatorCount ) that you need.

  AI Platform Training only supports distributed training when you train with TensorFlow or use a custom container . If your training job runs scikit-learn or XGBoost code, do not specify workers, parameter servers, or evaluators.

• Set the machine type for your master worker ( masterType ). If you have chosen to use workers, parameter servers, or evaluators, then set machine types for them in the workerType , parameterServerType , and evaluatorType fields respectively.

  You can specify different machine types for masterType , workerType , parameterServerType , and evaluatorType , but you can't use different machine types for individual instances. For example, you can use a n1-highmem-8 machine type for your parameter servers, but you can't set some parameter servers to use n1-highmem-8 and some to use n1-highcpu-16 .

• If you need just one worker with a custom configuration (not a full cluster), you should specify a custom scale tier with a machine type for the master only. That gives you just the single worker. Here's an example config.yaml file:

   trainingInput:
    scaleTier: CUSTOM
    masterType: n1-highcpu-16 
  

Compute Engine machine types

You can use the names of certain Compute Engine predefined machine types for your job's masterType , workerType , parameterServerType , and evaluatorType . If you are training with TensorFlow or using custom containers, you can optionally use various types of GPUs with these machine types.

The following list contains the Compute Engine machine type identifiers that you can use for your training job:

To learn about the technical specifications of each machine type, read the Compute Engine documentation about machine types .

Legacy machine types

Instead of using Compute Engine machine types for your job, you can specify legacy machine type names. These machine types provide the same vCPU and memory resources as equivalent Compute Engine machine types, but they have additional configuration limitations:

• You cannot customize GPU usage using an acceleratorConfig . However, some legacy machine types include GPUs. See the following table.

• If your training job configuration uses multiple machines, you cannot mix Compute Engine machine types with legacy machine types. Your master worker, workers, parameter servers, and evaluators must all use machine types from one group or the other.

  For example, if you configure masterType to be n1-highcpu-32 (a Compute Engine machine type), you cannot set workerType to complex_model_m (a legacy machine type), but you can set it to n1-highcpu-16 (another Compute Engine machine type).

The following table describes the legacy machine types:

Training with GPUs and TPUs

Some scale tiers and legacy machine types include graphics processing units (GPUs). You can also attach your own choice of several GPUs if you use a Compute Engine machine type. To learn more, read about training with GPUs .

To perform training with Tensor Processing Units (TPUs), you must use the BASIC_TPU scale tier or the cloud_tpu machine type. The cloud_tpu machine type has special configuration options: you can use it together with either Compute Engine machine types or with legacy machine types, and you can configure it to use 8 TPU v2 cores or 8 TPU v3 cores. Read about how to use TPUs for your training job .

What's next

• Run a training job .