Usage

Learn how to run models and apply basic optimizations using the LiteRT-LM CLI.

Quick Start

Run the Gemma4 E2B model:

 litert-lm  
run  
 \ 
  
--from-huggingface-repo = 
litert-community/gemma-4-E2B-it-litert-lm  
 \ 
  
gemma-4-E2B-it.litertlm  
 \ 
  
--prompt = 
 "What is the capital of France?" 
 

  litert-lm 
 run 
 ` 
 - 
 -from-huggingface-repo 
 = 
 litert-community 
 / 
 gemma 
 - 
 4-E2B-it-litert-lm 
 ` 
 gemma 
 - 
 4-E2B-it 
 . 
 litertlm 
 ` 
 - 
 -prompt 
 = 
 "What is the capital of France?" 
 

Run a local model

 litert-lm  
run  
path/to/model.litertlm 

  litert-lm 
 run 
 path 
 \ 
 to 
 \ 
 model 
 . 
 litertlm 
 

GPU Acceleration

To accelerate inference using your device's GPU, use the --backend=gpu flag:

 litert-lm  
run  
 \ 
  
--from-huggingface-repo = 
litert-community/gemma-4-E2B-it-litert-lm  
 \ 
  
gemma-4-E2B-it.litertlm  
 \ 
  
--backend = 
gpu  
 \ 
  
--prompt = 
 "What is the capital of France?" 
 

  litert-lm 
 run 
 ` 
 - 
 -from-huggingface-repo 
 = 
 litert-community 
 / 
 gemma 
 - 
 4-E2B-it-litert-lm 
 ` 
 gemma 
 - 
 4-E2B-it 
 . 
 litertlm 
 ` 
 - 
 -backend 
 = 
 gpu 
 ` 
 - 
 -prompt 
 = 
 "What is the capital of France?" 
 

Multi-Token Prediction (MTP)

Multi-Token Prediction (MTP) is a performance optimization that significantly accelerates decode speeds. MTP is universally recommended for all tasks on GPU backends.

To enable MTP in the CLI, use the --enable-speculative-decoding=true flag:

 litert-lm  
run  
 \ 
  
--from-huggingface-repo = 
litert-community/gemma-4-E2B-it-litert-lm  
 \ 
  
gemma-4-E2B-it.litertlm  
 \ 
  
--backend = 
gpu  
 \ 
  
--enable-speculative-decoding = 
 true 
  
 \ 
  
--prompt = 
 "What is the capital of France?" 
 

  litert-lm 
 run 
 ` 
 - 
 -from-huggingface-repo 
 = 
 litert-community 
 / 
 gemma 
 - 
 4-E2B-it-litert-lm 
 ` 
 gemma 
 - 
 4-E2B-it 
 . 
 litertlm 
 ` 
 - 
 -backend 
 = 
 gpu 
 ` 
 - 
 -enable-speculative-decoding 
 = 
 true 
 ` 
 - 
 -prompt 
 = 
 "What is the capital of France?" 
 

Multi-Modality

The LiteRT-LM CLI supports running multimodal models with image and audio attachments.

Prerequisites

To use attachments, you must specify the appropriate backend for processing them:

• For images: Use the --vision-backend option.
• For audio: Use the --audio-backend option.

Supported backends typically include cpu and gpu .

Image Attachments

To run a model with an image attachment:

 litert-lm  
run  
<model-ref>  
--vision-backend = 
gpu  
--attachment = 
image.jpg  
--prompt = 
 "Describe this image." 
 

Audio Attachments

To run a model with an audio attachment:

 litert-lm  
run  
<model-ref>  
--audio-backend = 
gpu  
--attachment = 
audio.wav  
--prompt = 
 "Transcribe this audio." 
 

Multiple Attachments

You can attach multiple files by repeating the --attachment option:

 litert-lm  
run  
<model-ref>  
--audio-backend = 
cpu  
--vision-backend = 
gpu  
--attachment = 
audio.wav  
--attachment = 
image.jpg  
... 

Function Calling

Augment your local LLMs with Python capabilities by running tools through presets.

Using Presets for Tool Use

You can run tools with presets. Create a preset.py file to define your tools and system instructions:

  import 
  
 datetime 
 def 
  
 get_current_time 
 () 
 - 
> str 
 : 
  
 """Returns the current date and time.""" 
 return 
 datetime 
 . 
 datetime 
 . 
 now 
 () 
 . 
 strftime 
 ( 
 "%Y-%m- 
 %d 
 %H:%M:%S" 
 ) 
 system_instruction 
 = 
 "You are a helpful assistant with access to tools." 
 tools 
 = 
 [ 
 get_current_time 
 ] 
 

Run the model with the preset:

 litert-lm  
run  
<model-ref>  
--preset = 
preset.py 

How it Works

When you ask a question that requires external information (like the current time), the model recognizes that it needs to call a tool:

1. Model Emits tool_call : The model outputs a JSON request to call the get_current_time function.
2. CLI Executes Tool: The LiteRT-LM CLI intercepts this call and executes the corresponding Python function defined in your preset.py .
3. CLI Sends tool_response : The CLI sends the result back to the model.
4. Model Generates Final Answer: The model uses the tool response to compute and generate the final answer for the user.

Sample interactive session:

 > what will the time be in two hours?
[tool_call] {"arguments": {}, "name": "get_current_time"}
[tool_response] {"name": "get_current_time", "response": "2026-03-25 21:54:07"}
The current time is 2026-03-25 21:54:07.

In two hours, it will be **2026-03-25 23:54:07**. 

This "Function Calling" loop happens automatically within the CLI, allowing you to augment local LLMs with Python capabilities without writing any complex orchestration code.