This page explains how to create and manage Spot VMs , including the following:
- How to create, start, and identify Spot VMs
- How to detect, handle, and test preemption of Spot VMs
- Best practices for Spot VMs
Spot VMs are virtual machine (VM) instances with the spot provisioning model . Spot VMs are available at a discount of up to 91% off of the default price of standard VMs. However, Compute Engine might reclaim the resources by preempting Spot VMs at any time. Spot VMs are recommended only for fault-tolerant workloads that can withstand VM preemption. Make sure your workload can manage preemption before you decide to create Spot VMs.
If you want to create and manage Spot VMs with TPUs, then see the Cloud TPU documentation for Spot VMs instead.
Before you begin
- Read the conceptual documentation for Spot VMs :
- Review the limitations and pricing of Spot VMs.
- To prevent Spot VMs from consuming your quotas for standard VMs' CPUs, GPUs, and disks, consider requesting preemptible quota for Spot VMs.
- If you haven't already, set up authentication
.
Authentication verifies your identity for access to Google Cloud services and APIs. To run
code or samples from a local development environment, you can authenticate to
Compute Engine by selecting one of the following options:
Select the tab for how you plan to use the samples on this page:
Console
When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.
gcloud
-
Install the Google Cloud CLI. After installation, initialize the Google Cloud CLI by running the following command:
gcloud init
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .
- Set a default region and zone .
Terraform
To use the Terraform samples on this page in a local development environment, install and initialize the gcloud CLI, and then set up Application Default Credentials with your user credentials.
-
Install the Google Cloud CLI.
-
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .
-
If you're using a local shell, then create local authentication credentials for your user account:
gcloud auth application-default login
You don't need to do this if you're using Cloud Shell.
If an authentication error is returned, and you are using an external identity provider (IdP), confirm that you have signed in to the gcloud CLI with your federated identity .
For more information, see Set up authentication for a local development environment .
REST
To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.
Install the Google Cloud CLI.
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .
For more information, see Authenticate for using REST in the Google Cloud authentication documentation.
-
Create a Spot VM
A Spot VM is any VM that is configured to use the spot provisioning model:
- VM provisioning modelset to Spotin the Google Cloud console
-
--provisioning-model=SPOTin the gcloud CLI -
"provisioningModel": "SPOT"in the Compute Engine API
To learn how to create a Spot VM, select one of the following methods based on how you want to handle preemption :
- Create a Spot VM with 120 seconds for handling preemption ( Preview )
- Create a Spot VM with up to 30 seconds for handling preemption (default)
Alternatively, to create multiple Spot VMs with the same properties, you can create an instance template , and use the template to create a managed instance group (MIG) . For more information, see best practices .
Create a Spot VM with 120 seconds for handling preemption
Before creating a Spot VM with 120-second preemption notice, make sure you handle preemption within your workload .
gcloud
To create a Spot VM with a 120-second preemption notice
duration from the gcloud CLI, use the gcloud beta compute instances create
command
and include the --preemption-notice-duration=120s
flag. To create
Spot VMs, you must include the --provisioning-model=SPOT
flag.
Optionally, you can specify a termination action for
Spot VMs by also including the --instance-termination-action
flag.
gcloud beta compute instances create VM_NAME \ --provisioning-model=SPOT \ --preemption-notice-duration=120s \ --instance-termination-action= TERMINATION_ACTION
Replace the following:
-
VM_NAME: name of the new VM. -
TERMINATION_ACTION: Optional: specify which termination action to take when Compute Engine preempts the VM, eitherSTOP(default behavior) orDELETE.
For more information about the options you can specify when creating a VM, see Configuration options during instance creation . For example, to create Spot VMs with a specified machine type and image, use the following command:
gcloud beta compute instances create VM_NAME \ --provisioning-model=SPOT \ --preemption-notice-duration=120s \ [--image= IMAGE | --image-family= IMAGE_FAMILY ] \ --image-project= IMAGE_PROJECT \ --machine-type= MACHINE_TYPE \ --instance-termination-action= TERMINATION_ACTION
Replace the following:
-
VM_NAME: name of the new VM. -
IMAGE: specify one of the following:-
IMAGE: a specific version of a public image or the image family. For example, a specific image is--image=debian-10-buster-v20200309. - An image family
.
This creates the VM from the most recent, non-deprecated OS image.
For example, if you specify
--image-family=debian-10, Compute Engine creates a VM from the latest version of the OS image in the Debian 10 image family.
-
-
IMAGE_PROJECT: the project containing the image. For example, if you specifydebian-10as the image family, specifydebian-cloudas the image project. -
MACHINE_TYPE: the predefined or custom machine type for the new VM.To get a list of the machine types available in a zone, use the
gcloud compute machine-types listcommand with the--zonesflag. -
TERMINATION_ACTION: Optional: specify which termination action to take when Compute Engine preempts the VM, eitherSTOP(default behavior) orDELETE.
REST
To create a Spot VM with a 120-second preemption notice
duration from the Compute Engine API, use the beta instances.insert
method
and include the "preemptionNoticeDuration": { "seconds": 120 }
field.
You must specify a machine type and name for the VM. Optionally, you can
also specify an image for the boot disk.
To create Spot VMs, you must include the "provisioningModel": spot
field.
Optionally, you can also specify a termination action for Spot VMs by also
including the "instanceTerminationAction"
field.
POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/instances { "machineType": "zones/ ZONE /machineTypes/ MACHINE_TYPE ", "name": "VM_NAME", "disks": [ { "initializeParams": { "sourceImage": "projects/ IMAGE_PROJECT /global/images/ IMAGE " }, "boot": true } ], "scheduling": { "provisioningModel": "SPOT", "preemptionNoticeDuration": { "seconds": 120 }, "instanceTerminationAction": " TERMINATION_ACTION " } }
Replace the following:
-
PROJECT_ID: the project ID of the project to create the VM in. -
ZONE: the zone to create the VM in. The zone must also support the machine type to use for the new VM. -
MACHINE_TYPE: the predefined or custom machine type for the new VM. -
VM_NAME: the name of the new VM. -
IMAGE_PROJECT: the project containing the image. For example, if you specifyfamily/debian-10as the image family, specifydebian-cloudas the image project. -
IMAGE: specify one of the following:- A specific version of a public image. For example, a specific image is
"sourceImage": "projects/debian-cloud/global/images/debian-10-buster-v20200309"wheredebian-cloudis theIMAGE_PROJECT. - An image family
.
This creates the VM from the most recent, non-deprecated OS image.
For example, if you specify
"sourceImage": "projects/debian-cloud/global/images/family/debian-10"wheredebian-cloudis theIMAGE_PROJECT, Compute Engine creates a VM from the latest version of the OS image in the Debian 10 image family.
- A specific version of a public image. For example, a specific image is
-
TERMINATION_ACTION: Optional: specify which termination action to take when Compute Engine preempts the VM, eitherSTOP(default behavior) orDELETE.
For more information about the options you can specify when creating a VM, see Configuration options during instance creation .
Create a Spot VM with up to 30 seconds for handling preemption (default)
By default, a Spot VM is notified of preemption at the start of 30-second best-effort shutdown period, and you must handle preemption within a shutdown script .
Console
-
In the Google Cloud console, go to the Create an instancepage.
-
In Machine configurationpane, which is open by default, complete the following steps:
- In the Provisioning modelsection, select Spotfrom the VM provisioning modellist.
-
Optional: To select the termination action that happens when Compute Engine preempts the VM, complete the following steps:
- Expand the VM provisioning model advanced settingssection.
- In the On VM terminationlist, select one of the following
options:
- To stop the VM during preemption, select Stop(default).
- To delete the VM during preemption, select Delete.
-
In the navigation menu, click Advanced. In the Advancedpane that appears, complete the following steps:
- In the Metadatasection, click Add item.
- In the Keyfield, enter
shutdown-scriptfor the metadata key. - In the Valuefield, add the contents of a shutdown script that handles preemption. For an example shutdown script, see Handle preemption in this document.
-
Optional: Specify other configuration options. For more information, see Configuration options during instance creation .
-
To create and start the VM, click Create.
gcloud
To create a VM from the gcloud CLI, use the gcloud compute instances create
command
.
To create Spot VMs, you must include the --provisioning-model=SPOT
flag. To handle preemption, include the --metadata shutdown-script
flag to specify a shutdown script. Optionally,
you can specify a termination action for Spot VMs by also
including the --instance-termination-action
flag.
gcloud compute instances create VM_NAME \ --provisioning-model=SPOT \ --metadata shutdown-script= SHUTDOWN_SCRIPT \ --instance-termination-action= TERMINATION_ACTION
Replace the following:
-
VM_NAME: name of the new VM. -
SHUTDOWN_SCRIPT: a shutdown script, which you can format as shown in Run shutdown scripts . For an example shutdown script, see Handle preemption in this document. -
TERMINATION_ACTION: Optional: specify which termination action to take when Compute Engine preempts the VM, eitherSTOP(default behavior) orDELETE.
For more information about the options you can specify when creating a VM, see Configuration options during instance creation . For example, to create Spot VMs with a specified machine type and image, use the following command:
gcloud compute instances create VM_NAME \ --provisioning-model=SPOT \ --metadata shutdown-script= SHUTDOWN_SCRIPT \ [--image= IMAGE | --image-family= IMAGE_FAMILY ] \ --image-project= IMAGE_PROJECT \ --machine-type= MACHINE_TYPE \ --instance-termination-action= TERMINATION_ACTION \
Replace the following:
-
VM_NAME: name of the new VM. -
SHUTDOWN_SCRIPT: a shutdown script that handles preemption. For more information about how to format and specify a shutdown script, see Run shutdown scripts . For an example shutdown script, see Handle preemption in this document. -
IMAGE: specify one of the following:-
IMAGE: a specific version of a public image or the image family. For example, a specific image is--image=debian-10-buster-v20200309. - An image family
.
This creates the VM from the most recent, non-deprecated OS image.
For example, if you specify
--image-family=debian-10, Compute Engine creates a VM from the latest version of the OS image in the Debian 10 image family.
-
-
IMAGE_PROJECT: the project containing the image. For example, if you specifydebian-10as the image family, specifydebian-cloudas the image project. -
MACHINE_TYPE: the predefined or custom machine type for the new VM.To get a list of the machine types available in a zone, use the
gcloud compute machine-types listcommand with the--zonesflag. -
TERMINATION_ACTION: Optional: specify which termination action to take when Compute Engine preempts the VM, eitherSTOP(default behavior) orDELETE.
Terraform
You can use a Terraform
resource
to create a Spot VM using the scheduling
block as shown in
the following example.
To add a shutdown script for handling preemption, also add a metadata
block as shown in Run shutdown scripts
.
For an example shutdown script, see Handle preemption
in this document.
REST
To create a VM from the Compute Engine API, use the instances.insert
method
.
You must specify a machine type and name for the VM. Optionally, you can
also specify an image for the boot disk.
To create Spot VMs, you must include the "provisioningModel": spot
field.
To handle preemption, also include the "metadata"
field and specify a shutdown script.
Optionally, you can specify a termination action for Spot VMs by also
including the "instanceTerminationAction"
field.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances { "machineType": "zones/ ZONE /machineTypes/ MACHINE_TYPE ", "name": "VM_NAME", "disks": [ { "initializeParams": { "sourceImage": "projects/ IMAGE_PROJECT /global/images/ IMAGE " }, "boot": true } ], "scheduling": { "provisioningModel": "SPOT", "instanceTerminationAction": " TERMINATION_ACTION " }, "metadata": { "items": [ { "key": "shutdown-script", "value": " SHUTDOWN_SCRIPT " } ] } }
Replace the following:
-
PROJECT_ID: the project ID of the project to create the VM in. -
ZONE: the zone to create the VM in. The zone must also support the machine type to use for the new VM. -
MACHINE_TYPE: the predefined or custom machine type for the new VM. -
VM_NAME: the name of the new VM. -
IMAGE_PROJECT: the project containing the image. For example, if you specifyfamily/debian-10as the image family, specifydebian-cloudas the image project. -
IMAGE: specify one of the following:- A specific version of a public image. For example, a specific image is
"sourceImage": "projects/debian-cloud/global/images/debian-10-buster-v20200309"wheredebian-cloudis theIMAGE_PROJECT. - An image family
.
This creates the VM from the most recent, non-deprecated OS image.
For example, if you specify
"sourceImage": "projects/debian-cloud/global/images/family/debian-10"wheredebian-cloudis theIMAGE_PROJECT, Compute Engine creates a VM from the latest version of the OS image in the Debian 10 image family.
- A specific version of a public image. For example, a specific image is
-
TERMINATION_ACTION: Optional: specify which termination action to take when Compute Engine preempts the VM, eitherSTOP(default behavior) orDELETE. -
SHUTDOWN_SCRIPT: a shutdown script that handles preemption. For more information about how to format and specify a shutdown script, see Run shutdown scripts . For an example shutdown script, see Handle preemption in this document.
For more information about the options you can specify when creating a VM, see Configuration options during instance creation .
Go
The following example creates a Spot VM without a shutdown script for handling preemption. For more information about how to specify a shutdown script for a new or existing VM, see Run shutdown scripts . For an example shutdown script, see Handle preemption in this document.
Java
The following example creates a Spot VM without a shutdown script for handling preemption. For more information about how to specify a shutdown script for a new or existing VM, see Run shutdown scripts . For an example shutdown script, see Handle preemption in this document.
Python
The following example creates a Spot VM without a shutdown script for handling preemption. For more information about how to specify a shutdown script for a new or existing VM, see Run shutdown scripts . For an example shutdown script, see Handle preemption in this document.
Start Spot VMs
Like other VMs, Spot VMs start upon creation. Likewise, if
Spot VMs are stopped, you can restart the VMs
to resume the RUNNING
state.
You can stop and restart preempted Spot VMs
as many times as you would like, as long as there is capacity.
For more information, see VM instance life cycle
.
If Compute Engine stops one or more Spot VMs in an autoscaling managed instance group (MIG) or Google Kubernetes Engine (GKE) cluster, the group restarts the VMs when the resources become available again.
Identify a VM's provisioning model and termination action
Identify a VM's provisioning model to see if it is a standard VM, Spot VM, or preemptible VM . For a Spot VM, you can also identify the termination action . You can identify a VM's provisioning model and termination action using the Google Cloud console, gcloud CLI, or the Compute Engine API.
Console
-
Go to the VM instancespage.
-
Click the Nameof the VM you want to identify. The VM instance detailspage opens.
-
Go to the Managementsection at the bottom of the page. In the Availability policiessubsection, check the following options:
- If the VM provisioning modelis set to Spot, the VM is a
Spot VM.
- On VM terminationindicates which action to take when Compute Engine preempts the VM, either Stopor Deletethe VM.
- Otherwise, if the VM provisioning modelis set to Standardor —:
- If the Preemptibilityoption is set to On, the VM is a preemptible VM.
- Otherwise, the VM is a standard VM.
- If the VM provisioning modelis set to Spot, the VM is a
Spot VM.
gcloud
To describe a VM from the gcloud CLI, use the gcloud compute instances describe
command
:
gcloud compute instances describe VM_NAME
where VM_NAME
is the name
of the VM
that you want to check.
In the output, check the scheduling
field to identify the VM:
-
If the output includes the
provisioningModelfield set toSPOT, similar to the following, the VM is a Spot VM.... scheduling: ... provisioningModel: SPOT instanceTerminationAction: TERMINATION_ACTION ...
where
TERMINATION_ACTIONindicates which action to take when Compute Engine preempts the VM, either stop (STOP) or delete (DELETE) the VM. If theinstanceTerminationActionfield is missing, the default value isSTOP. -
Otherwise, if the output includes the
provisioningModelfield set tostandardor if the output omits theprovisioningModelfield:- If the output includes the
preemptiblefield set totrue, the VM is a preemptible VM. - Otherwise, the VM is a standard VM.
- If the output includes the
REST
To describe a VM from the Compute Engine API, use the instances.get
method
:
GET https://compute.googleapis.com/compute/v1/projects/ PROJECT_ID /zones/ ZONE /instances/ VM_NAME
Replace the following:
-
PROJECT_ID: the project ID of the project that the VM is in. -
ZONE: the zone where the VM is located. -
VM_NAME: the name of the VM that you want to check.
In the output, check the scheduling
field to identify the VM:
-
If the output includes the
provisioningModelfield set toSPOT, similar to the following, the VM is a Spot VM.{ ... "scheduling": { ... "provisioningModel": "SPOT", "instanceTerminationAction": " TERMINATION_ACTION " ... }, ... }
where
TERMINATION_ACTIONindicates which action to take when Compute Engine preempts the VM, either stop (STOP) or delete (DELETE) the VM. If theinstanceTerminationActionfield is missing, the default value isSTOP. -
Otherwise, if the output includes the
provisioningModelfield set tostandardor if the output omits theprovisioningModelfield:- If the output includes the
preemptiblefield set totrue, the VM is a preemptible VM. - Otherwise, the VM is a standard VM.
- If the output includes the
Go
Java
Python
Manage preemption of a Spot VM
To learn how to manage the preemption of a Spot VM, review the following sections:
Handle preemption
When Compute Engine begins preempting a Spot VM, you can try to perform cleanup actions before the VM finishes shutting down. Handling preemption can include gracefully stopping a running process and transferring the state of your workload.
You can use the following methods to handle preemption for a Spot VM:
- Handle preemption within your workload.We recommend this method for Spot VMs with a 120-second preemption notice duration ( Preview ). Specifically, within your workload, configure code for handling preemption to wait to run until preemption starts as explained in Detect preemption within a VM . Then, your code for handling preemption runs during the preemption notice duration . (Optionally, these VMs can also specify a shutdown script, which runs during a shutdown period.)
- Handle preemption within a shutdown script.We recommend this method for Spot VMs without a preemption notice duration, which is the default configuration. Specifically, configure your code for handling preemption within a shutdown script as shown in the following example. The shutdown script automatically runs for up to 30 seconds during the best-effort shutdown period for any type of shutdown. Consequently, you might want to configure the code for handling preemption to only run if the VM is being preempted as explained in Detect preemption within a VM .
The following example demonstrates how to handle preemption by uploading a
checkpoint file to a Cloud Storage bucket
within a
shutdown script, and how to add a shutdown script while creating or updating a
Spot VM. This script runs when the VM starts to shut down,
before the operating system's normal kill
command stops all remaining
processes. After gracefully stopping a specified program, the script performs a
parallel upload of a checkpoint file to a Cloud Storage bucket.
#!/bin/bash MY_PROGRAM=" PROGRAM_NAME " # For example, "apache2" or "nginx" MY_USER=" LOCAL_USER " CHECKPOINT="/home/$MY_USER/checkpoint.out" BUCKET_NAME=" BUCKET_NAME " # For example, "my-checkpoint-files" (without gs://) echo "Shutting down! Seeing if ${MY_PROGRAM} is running." # Find the newest copy of $MY_PROGRAM PID="$(pgrep -n "$MY_PROGRAM")" if [[ "$?" -ne 0 ]]; then echo "${MY_PROGRAM} not running, shutting down immediately." exit 0 fi echo "Sending SIGINT to $PID" kill -2 "$PID" # Portable waitpid equivalent while kill -0 "$PID"; do sleep 1 done echo "$PID is done, copying ${CHECKPOINT} to gs://${BUCKET_NAME} as ${MY_USER}" su "${MY_USER}" -c "gcloud storage cp $CHECKPOINT gs://${BUCKET_NAME}/" echo "Done uploading, shutting down."
This script assumes the following:
-
The VM was created with at least read/write access to Cloud Storage. For instructions about how to create a VM with the appropriate scopes, see the authentication documentation .
-
You have an existing Cloud Storage bucket and permission to write to it.
To add this script to a VM, configure the script to work with the workload on your VM and add it to the VM's metadata.
-
Copy or download the shutdown script:
-
Copy the preceding shutdown script after replacing the following:
-
PROGRAM_NAMEis the name of the process or program you want to shut down. For example,apache2ornginx. -
LOCAL_USERis the username you are logged into the virtual machine as. -
BUCKET_NAMEis the name of the Cloud Storage bucket where you want to save the program's checkpoint file. Note the bucket name does not start withgs://in this case.
-
-
Download the shutdown script to your local workstation and then replace the following variables in the file:
-
[PROGRAM_NAME]is the name of the process or program you want to shut down. For example,apache2ornginx. -
[LOCAL_USER]is the username you are logged into the virtual machine as. -
[BUCKET_NAME]is the name of the Cloud Storage bucket where you want to save the program's checkpoint file. Note the bucket name does not start withgs://in this case.
-
-
-
Add the shutdown script to a new VM or an existing VM .
Detect preemption of Spot VMs
The following sections explain the methods you can use to detect preemption of Spot VMs.
- Detect preemption within a VM : For example, use this method to check for preemption within a shutdown script or to trigger preemption handling for Spot VMs with a 120-second preemption notice duration ( Preview ).
- View preemption operations : For example, use this method when you want to determine why Spot VMs shut down.
Detect preemption within a VM
To detect if a VM is being preempted from inside the VM itself, check the
metadata server for the preempted
value in your VM's default metadata
. For example, use the
following methods:
-
Check the current value of
preempted.You can run the followingcurlcommand from within your VM to obtain the current value forpreempted:curl "http://metadata.google.internal/computeMetadata/v1/instance/preempted" -H "Metadata-Flavor: Google" TRUE
If this value is
TRUE, the VM was preempted by Compute Engine, otherwise it isFALSE. For example, use this command within a shutdown script to check if the shutdown was caused by preemption or not. -
Wait until
preemptedisTRUE.To wait untilpreemptedisTRUE, you can append?wait_for_change=trueto the URL of the previous command. This command performs a hanging HTTP GET request that only returns when the metadata has changed and the VM has been preempted.curl "http://metadata.google.internal/computeMetadata/v1/instance/preempted?wait_for_change=true" -H "Metadata-Flavor: Google" TRUE
This command is useful when you want to trigger preemption handling outside of a shutdown script. For example, use this method to trigger preemption handling for Spot VMs with a 120-second preemption notice duration ( Preview ).
View preemption operations
You can view preemption operations from Compute Engine by using the Google Cloud console , gcloud CLI or the Compute Engine API .
Console
You can check if a VM was preempted by checking the system activity logs.
-
In the Google Cloud console, go to the Logspage.
-
Select your project and click Continue.
-
Add
compute.instances.preemptedto the filter by label or text searchfield. -
Optionally, you can also enter a VM name if you want to see preemption operations for a specific VM.
-
Press enter to apply the specified filters. The Google Cloud console updates the list of logs to show only the operations where a VM was preempted.
-
Select an operation in the list to see details about the VM that was preempted.
gcloud
Use the gcloud compute operations list
command
with a filter
parameter to
get a list of preemption events in your project.
gcloud compute operations list \
--filter="operationType=compute.instances.preempted"
Optionally, you can use additional filter parameters to further scope the results. For example, to see preemption events only for instances within a managed instance group, use the following command:
gcloud compute operations list \ --filter="operationType=compute.instances.preempted AND targetLink:instances/ BASE_INSTANCE_NAME "
where BASE_INSTANCE_NAME
is the base name
specified as a prefix for the names of all the VMs in this
managed instance group.
The output is similar to the following:
NAME TYPE TARGET HTTP_STATUS STATUS TIMESTAMP systemevent-xxxxxxxx compute.instances.preempted us-central1-f/instances/example-instance-xxx 200 DONE 2015-04-02T12:12:10.881-07:00
An operation type of compute.instances.preempted
indicates that the
VM instance was preempted. You can use the gcloud compute operations describe
command
to get more information about a specific preemption operation.
gcloud compute operations describe SYSTEM_EVENT \ --zone= ZONE
Replace the following:
-
SYSTEM_EVENT: the system event from the output of thegcloud compute operations listcommand—for example,systemevent-xxxxxxxx. -
ZONE: the zone of the system event—for example,us-central1-f.
The output is similar to the following:
... operationType: compute.instances.preempted progress: 100 selfLink: https://compute.googleapis.com/compute/v1/projects/my-project/zones/us-central1-f/operations/systemevent-xxxxxxxx startTime: '2015-04-02T12:12:10.881-07:00' status: DONE statusMessage: Instance was preempted. ...
REST
To get a list of recent system operations for a specific project and zone,
use the zoneOperations.get
method
.
GET https://compute.googleapis.com/compute/v1/projects/ PROJECT_ID /zones/ ZONE /operations
Replace the following:
-
PROJECT_ID: a project ID . -
ZONE: a zone .
Optionally, to scope the response to show only preemption operations, you can add a filter to your API request:
operationType="compute.instances.preempted"
Alternatively, to see preemption operations
for a specific VM, add a targetLink
parameter to the filter:
operationType="compute.instances.preempted" AND targetLink="https://www.googleapis.com/compute/v1/projects/ PROJECT_ID /zones/ ZONE /instances/ VM_NAME
Replace the following:
+ PROJECT_ID
: the project ID
.
+ ZONE
: the zone
.
+ VM_NAME
: the name of a specific VM in this
zone and project.
The response contains a list of recent operations. For example, a preemption looks similar to the following:
{
"kind": "compute#operation",
"id": "15041793718812375371",
"name": "systemevent-xxxxxxxx",
"zone": "https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-f",
"operationType": "compute.instances.preempted",
"targetLink": "https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-f/instances/example-instance",
"targetId": "12820389800990687210",
"status": "DONE",
"statusMessage": "Instance was preempted.",
...
}
Test preemption settings
You can run simulated maintenance events on your Spot VMs to force them to preempt. Use this feature to test how your workloads detect and handle preemption. To learn how to test maintenance events on your instances, see Simulate a host maintenance event .
Best practices
Here are some best practices to help you get the most out of Spot VMs.
-
Use instance templates. Rather than creating Spot VMs one at a time, you can use instance templates to create multiple Spot VMs with the same properties. Instance templates are required for using MIGs. Alternatively, you can also create multiple Spot VMs using the bulk instance API .
-
Use MIGs to regionally distribute and automatically recreate Spot VMs. Use MIGs to make workloads on Spot VMs more flexible and resilient. For example, use regional MIGs to distribute VMs across multiple zones, which helps mitigate resource-availability errors. Additionally, use autohealing to automatically recreate Spot VMs after they are preempted.
-
Pick smaller machine types. Resources for Spot VMs come out of excess and backup Google Cloud capacity. Capacity for Spot VMs is often easier to get for smaller machine types , meaning machine types with less resources like vCPUs and memory. You might find more capacity for Spot VMs by selecting a smaller custom machine type, but capacity is even more likely for smaller predefined machine types. For example, compared to capacity for the
n2-standard-32predefined machine type, capacity for then2-custom-24-96custom machine type is more likely, but capacity for then2-standard-16predefined machine type is even more likely. -
Run large clusters of Spot VMs during off peak times. The load on Google Cloud data centers varies with location and time of day, but generally lowest on nights and weekends. As such, nights and weekends are the best times to run large clusters of Spot VMs.
-
Design your workloads to be fault and preemption tolerant. It's important to be prepared for the fact that there are changes in preemption patterns at different points in time. For example, if a zone suffers a partial outage, large numbers of Spot VMs could be preempted to make room for standard VMs that need to be moved as part of the recovery. In that small window of time, the preemption rate would look very different than on any other day. If your workload assumes that preemptions are always done in small groups, you might not be prepared for such an event.
-
Retry creating Spot VMs that have been preempted. If your Spot VMs have been preempted, try creating new Spot VMs once or twice before falling back to standard VMs. Depending on your requirements, it might be a good idea to combine standard VMs and Spot VMs in your clusters to ensure that work proceeds at an adequate pace.
-
Use shutdown scripts. Manage shutdown and preemption notices with a shutdown script that can save a job's progress so that it can pick up where it left off, rather than start over from scratch.
What's next?
- Connect to your VM instance .
- Learn about shutdown scripts .
- Learn about limiting the runtime of a VM .
- Learn about instance templates .
- Learn about MIGs .

