Get started with the ML Diagnostics CLI

Use the ML Diagnostics Google Cloud CLI to create a machine learning run, deploy XProf as a managed instance with a scalable backend, and provide a managed profiling experience on Google Cloud.

There are two categories of ML Diagnostics gcloud CLI commands: machine-learning-run commands and profiler commands. Use the machine-learning-run commands to create, delete, describe, list, and update machine learning runs. Use the profiler commands to list nodes and capture on-demand profiles from the CLI.

• Machine-learning-run commands: Create , Delete , Describe , List , Update .
• Profiler commands:
  • profiler-target : List
  • profiler-session : Capture , List

All gcloud CLI commands require a project defined in the environment. To set the project:

 gcloud  
config  
 set 
  
project  
 PROJECT_ID 
 

For more information on the ML Diagnostics gcloud CLI commands, see the API reference .

Capture profiles

You can capture XProf profiles of your ML workload with programmatic capture or on-demand capture (manual capture). Programmatic capture involves embedding profiling commands directly into your machine learning code, and explicitly stating when to start and stop recording data. On-demand capture occurs in real-time, where you trigger the profiler while the workload is already actively running.

To enable on-demand profile capture, you need to start the XProf server within your code and call the profiler.start_server method. This starts an XProf server on your ML workload that listens for the on-demand capture trigger to start capturing profiles. Use port 9999 for this command: profiler.start_server(port=9999)

For both programmatic and on-demand profile capture, specify the location to store the captured profiles. For example: gs://my-bucket/my-run . Profiles are stored in directories nested within the location: gs://my-bucket/my-run/plugins/profile/session1/ . Programmatic profile capture and on-demand capture must not occur during the same time period.

For on-demand profile capture, set up a GKE cluster and deploy workload with the label: managed-mldiagnostics-gke=true .

For more information about profiling with JAX, see Profiling computation .

Create machine learning run

Create a machine learning run resource in a specified project and location. The machine-learning-run create command deploys XProf as a managed instance in your project. The managed XProf instance is used for viewing all profiles in the project, and is created when the first machine learning run is created in the project.

Use the machine-learning-run create command:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
create 

There are two ways to create a machine learning run:

• Register existing captured profiles to the ML Diagnostics platform.
• Use ML Diagnostics to perform on-demand profile capture by registering an active run. This requires a GKE cluster setup and a deployed workload on GKE with the label managed-mldiagnostics-gke=true .

Create ML Run and register existing captured profiles

The following code creates a run and registers existing captured profiles to ML Diagnostics:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
create  
 RUN_NAME 
  
 \ 
  
--location  
 LOCATION 
  
 \ 
  
--run-group  
 GROUP_NAME 
  
 \ 
  
--gcs-path  
gs:// BUCKET_NAME 
  
 \ 
  
--display-name  
 DISPLAY_NAME 
  
 \ 
  
--labels  
 "list_existing_sessions_only" 
 = 
 "true" 
 

The code example uses the following flags:

The --labels list_existing_sessions_only=true flag is required if you want to view and manage existing collected profiles in ML Diagnostics. The flag does the following:

1. Creates a machine learning run with state "Completed".
2. Recursively searches for xplane.pb files within the Cloud Storage directory path.
3. Loads all located profile sessions into the ML Diagnostics database to view in Google Cloud, creates shareable links for the profile sessions, and allows users to manage these profiles with ML Diagnostics platform.

If the --labels list_existing_sessions_only flag is set to true for a run, you cannot perform on-demand profiling or update the run. You can only view and manage existing profiles.

Create ML Run to perform on-demand profile capture

The following code creates an mlrun in order perform on-demand profile capture:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
create  
 RUN_NAME 
  
 \ 
  
--location  
 LOCATION 
  
 \ 
  
--orchestrator  
gke  
 \ 
  
--run-group  
 RUN_GROUP 
  
 \ 
  
--gcs-path  
gs:// BUCKET_NAME 
  
 \ 
  
--display-name  
 DISPLAY_NAME 
  
 \ 
  
--gke-cluster-name  
projects/user/locations/ LOCATION 
/clusters/ CLUSTER_NAME 
  
 \ 
  
--gke-namespace  
 NAMESPACE 
  
 \ 
  
--gke-workload-name  
 WORKLOAD_NAME 
  
 \ 
  
--gke-kind  
 GKE_KIND 
  
 \ 
  
--gke-workload-create-time  
 CREATE_TIME 
  
 \ 
  
--run-phase  
 RUN_PHASE 
 

Along with the flags from the previous example, the code example uses the following additional flags:

Describe machine learning run

View the details of a machine learning run with the machine-learning-run describe command:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
describe  
 RUN_NAME 
  
--FORMAT = 
 FORMAT 
 

The following example is a request for run details in JSON:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
describe  
my-run-on-demand  
 \ 
  
--format  
json 

The output is similar to the following:

  { 
  
 "artifacts" 
 : 
  
 { 
  
 "gcsPath" 
 : 
  
 "gs://my-bucket" 
  
 }, 
  
 "createTime" 
 : 
  
 "2026-02-05T16:25:28.367865234Z" 
 , 
  
 "displayName" 
 : 
  
 "mldiagnostics-my-run-on-demand" 
 , 
  
 "endTime" 
 : 
  
 "0001-01-01T00:00:00Z" 
 , 
  
 "etag" 
 : 
  
 "1f54a7f4-bd25-4f98-a91c-97bfa1c5b7a6" 
 , 
  
 "name" 
 : 
  
 "projects/163028815180/locations/us-central1/machineLearningRuns/my-run-on-demand" 
 , 
  
 "orchestrator" 
 : 
  
 "GKE" 
 , 
  
 "runPhase" 
 : 
  
 "ACTIVE" 
 , 
  
 "runSet" 
 : 
  
 "my-run-on-demand-group" 
 , 
  
 "tools" 
 : 
  
 [ 
  
 { 
  
 "XProf" 
 : 
  
 {} 
  
 } 
  
 ], 
  
 "updateTime" 
 : 
  
 "2026-02-05T16:25:28.367865344Z" 
 , 
  
 "workloadDetails" 
 : 
  
 { 
  
 "gke" 
 : 
  
 { 
  
 "cluster" 
 : 
  
 "projects/163028815180/locations/us-central1/clusters/my-cluster" 
 , 
  
 "id" 
 : 
  
 "jobset-abcd" 
 , 
  
 "kind" 
 : 
  
 "JobSet" 
 , 
  
 "namespace" 
 : 
  
 "default" 
  
 } 
  
 } 
 } 
 

List machine learning runs

Get a list of machine learning runs within a specified project and location with the machine-learning-run list command:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
list 

The following example is a request for a list of up to two runs, with outputs of their URI paths:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
list  
--limit  
 2 
  
--uri 

 https://hypercomputecluster.googleapis.com/v1alpha/projects/163028815180/locations/us-central1/machineLearningRuns/my-run-on-demand
https://hypercomputecluster.googleapis.com/v1alpha/projects/163028815180/locations/us-central1/machineLearningRuns/my-run-on-demand-2 

Update machine learning runs

Update a machine learning run in a specified project and location. You can update the display name, run phase, orchestrator, and GKE workload details. You cannot change the run ID and location. Update a run with the machine-learning-run update command:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
update 

Provide all fields that were included in the create request . If mandatory fields are not provided during update request, they are overridden by the default values.

The etag flag is a mandatory field, and should be the latest ETag (entity tag) value for an ML Run resource. For more information, see Use entity tags for optimistic concurrency control . Use the following to find the correct ETAG value:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
describe  
 RUN_NAME 
 

The following is an example of a complete update request:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
update  
my-run-on-demand  
 \ 
  
--orchestrator  
gke  
 \ 
  
--run-group  
my-run-on-demand-group  
 \ 
  
--gcs-path  
gs://my-bucket  
 \ 
  
--display-name  
mldiagnostics-my-run-on-demand-completed  
 \ 
  
--gke-cluster-name  
projects/user/locations/us-central1/clusters/my-cluster  
 \ 
  
--gke-namespace  
default  
 \ 
  
--gke-workload-name  
jobset-abcd  
 \ 
  
--gke-kind  
JobSet  
 \ 
  
--gke-workload-create-time  
 2026 
-02-20T06:06:06Z  
 \ 
  
--run-phase  
COMPLETED  
 \ 
  
--etag  
1f54a7f4-bd25-4f98-a91c-97bfa1c5b7a6 

Delete machine learning runs

Delete a machine learning run in a specified project and location with the machine-learning-run delete command:

 gcloud  
alpha  
mldiagnostics  
machine-learning-run  
delete  
 RUN_NAME 
 

Deleting an ML run does not delete any data in Cloud Storage, Cloud Logging, or the GKE workload. Deleting the mlrun only deletes metadata related to the run within the ML Diagnostics system.

Profiler commands

You can use the profiler command group to list all profiles, find GKE nodes of workload where the XProf server is running, and capture on-demand profiles from the CLI.

List profiler targets

List all profiler targets associated with a machine learning run in a specified project and location:

 gcloud  
alpha  
mldiagnostics  
profiler-target  
list  
--machine-learning-run  
 RUN_NAME 
 

This command requires the following:

• On-demand Xprof is enabled in the workload, which deploys XProf server into all nodes of the workload.
• GKE cluster is set up for ML Diagnostics, with deployed webhook and operator.
• Deployed workload on GKE with the label: managed-mldiagnostics-gke=true .

The following is an example of a request:

 gcloud  
alpha  
mldiagnostics  
profiler-target  
list  
 \ 
  
--machine-learning-run  
my-run-on-demand 

The following is an example of the output:

 ---
hostname: gke-tpu-1f0789b5-jqx9
name: projects/163028815180/locations/us-central1/machineLearningRuns/my-run-on-demand/profilerTargets/jobset-abcd-tpu-slice-0-0-tcw2k
---
hostname: gke-tpu-1f0789b5-rxvf
name: projects/163028815180/locations/us-central1/machineLearningRuns/my-run-on-demand/profilerTargets/jobset-abcd-tpu-slice-0-1-dct59 

List profiler sessions

List all profiler sessions associated with a machine learning run in a specified project and location with the following command:

 gcloud  
alpha  
mldiagnostics  
profiler-session  
list  
--machine-learning-run  
 RUN_NAME 
 

This profiler command does not require GKE or workload setup. It will list all profile sessions, both programmatic and on-demand. If you only have programmatic profile captures, use this command to list all profile sessions. There is no required GKE setup, GKE workload labeling, or on-demand XProf enablement.

The following is an example of a request:

 gcloud  
alpha  
mldiagnostics  
profiler-session  
list  
 \ 
  
--machine-learning-run  
my-run-on-demand 

Capture on-demand profiler sessions

You can capture an on-demand profiler session for a machine learning run on a specified set of nodes that the workload is running on (profiler targets).

This command requires the following:

• On-demand XProf is enabled in the workload, which deploys XProf server into all nodes of the workload
• GKE cluster is set up for ML Diagnostics, with deployed webhook and operator
• Deployed workload on GKE with the label: managed-mldiagnostics-gke=true .

The following is an example of a request:

 gcloud  
alpha  
mldiagnostics  
profiler-session  
capture  
 \ 
  
profiler-session-on-demand  
 \ 
  
--machine-learning-run  
 RUN_NAME 
  
 \ 
  
--targets  
 TARGET 
  
 \ 
  
--duration  
 DURATION 
 

The example uses the following flags: