Introducing Google AI Edge Portal : Benchmark Edge AI at scale. Sign-up to request access during private preview.

Get Started with LiteRT-LM on Android

The Kotlin API of LiteRT-LM for Androidand JVM (Linux, macOS, Windows)with features like GPU and NPU acceleration, multi-modality, and tools use.

Introduction

Here is a sample terminal chat app built with the Kotlin API:

  import 
  
 com.google.ai.edge.litertlm.* 
 suspend 
  
 fun 
  
 main 
 () 
  
 { 
  
 Engine 
 . 
 setNativeMinLogSeverity 
 ( 
 LogSeverity 
 . 
 ERROR 
 ) 
  
 // Hide log for TUI app 
  
 val 
  
 engineConfig 
  
 = 
  
 EngineConfig 
 ( 
 modelPath 
  
 = 
  
 "/path/to/model.litertlm" 
 ) 
  
 Engine 
 ( 
 engineConfig 
 ). 
 use 
  
 { 
  
 engine 
  
 - 
>  
 engine 
 . 
 initialize 
 () 
  
 engine 
 . 
 createConversation 
 (). 
 use 
  
 { 
  
 conversation 
  
 - 
>  
 while 
  
 ( 
 true 
 ) 
  
 { 
  
 print 
 ( 
 "\n>>> " 
 ) 
  
 conversation 
 . 
 sendMessageAsync 
 ( 
 readln 
 ()). 
 collect 
  
 { 
  
 print 
 ( 
 it 
 ) 
  
 } 
  
 } 
  
 } 
  
 } 
 }

Demo for the Kotlin sample code

To try out the above sample, clone the repository and run with example/Main.kt :

  bazel 
 run 
 - 
 c 
 opt 
 // 
 kotlin 
 / 
 java 
 / 
 com 
 / 
 google 
 / 
 ai 
 / 
 edge 
 / 
 litertlm 
 / 
 example 
 : 
 main 
 -- 
< abs_model_path 
>

Available .litertlm models are on the HuggingFace LiteRT Community . The above animation was using the Gemma3-1B-IT .

For Android sample, check out the Google AI Edge Gallery app ( available on Google Play ).

Getting Started with Gradle

While LiteRT-LM is developed with Bazel, we provide the Maven packages for Gradle or Maven users.

Add the Gradle dependency

  dependencies 
  
 { 
  
 // For Android 
  
 implementation 
 ( 
 "com.google.ai.edge.litertlm:litertlm-android:latest.release" 
 ) 
  
 // For JVM (Linux, macOS, Windows) 
  
 implementation 
 ( 
 "com.google.ai.edge.litertlm:litertlm-jvm:latest.release" 
 ) 
 }

You can find the available versions on Google Maven in litertlm-android and litertlm-jvm .

latest.release can be used to get the latest release.

Initialize the Engine

The Engine is the entry point to the API. Initialize it with the model path and configuration. Remember to close the engine to release resources.

Note:The engine.initialize() method can take a significant amount of time (e.g., up to 10 seconds) to load the model. It is strongly recommended to call this on a background thread or coroutine to avoid blocking the UI thread.

  import 
  
 com.google.ai.edge.litertlm.Backend 
 import 
  
 com.google.ai.edge.litertlm.Engine 
 import 
  
 com.google.ai.edge.litertlm.EngineConfig 
 val 
  
 engineConfig 
  
 = 
  
 EngineConfig 
 ( 
  
 modelPath 
  
 = 
  
 "/path/to/your/model.litertlm" 
 , 
  
 // Replace with your model path 
  
 backend 
  
 = 
  
 Backend 
 . 
 GPU 
 (), 
  
 // Or Backend.NPU(nativeLibraryDir = "...") 
  
 // Optional: Pick a writable dir. This can improve 2nd load time. 
  
 // cacheDir = "/tmp/" or context.cacheDir.path (for Android) 
 ) 
 val 
  
 engine 
  
 = 
  
 Engine 
 ( 
 engineConfig 
 ) 
 engine 
 . 
 initialize 
 () 
 // ... Use the engine to create a conversation ... 
 // Close the engine when done 
 engine 
 . 
 close 
 ()

On Android, to use the GPU backend, the app needs to request the depending native libraries explicitly by adding the following to your AndroidManifest.xml inside the <application> tag:

   
<application>  
<uses-native-library  
android:name="libvndksupport.so"  
android:required="false"/>  
<uses-native-library  
android:name="libOpenCL.so"  
android:required="false"/>  
</application>

To use the NPUbackend, you might need to specify the directory containing the NPU libraries. On Android, if the libraries are bundled with your app, set it to context.applicationInfo.nativeLibraryDir . See LiteRT-LM NPU for more details about the NPU native libraries.

  val 
  
 engineConfig 
  
 = 
  
 EngineConfig 
 ( 
  
 modelPath 
  
 = 
  
 modelPath 
 , 
  
 backend 
  
 = 
  
 Backend 
 . 
 NPU 
 ( 
 nativeLibraryDir 
  
 = 
  
 context 
 . 
 applicationInfo 
 . 
 nativeLibraryDir 
 ) 
 )

Create a Conversation

Once the engine is initialized, create a Conversation instance. You can provide a ConversationConfig to customize its behavior.

  import 
  
 com.google.ai.edge.litertlm.ConversationConfig 
 import 
  
 com.google.ai.edge.litertlm.Message 
 import 
  
 com.google.ai.edge.litertlm.SamplerConfig 
 // Optional: Configure the system instruction, initial messages, sampling 
 // parameters, etc. 
 val 
  
 conversationConfig 
  
 = 
  
 ConversationConfig 
 ( 
  
 systemInstruction 
  
 = 
  
 Contents 
 . 
 of 
 ( 
 "You are a helpful assistant." 
 ), 
  
 initialMessages 
  
 = 
  
 listOf 
 ( 
  
 Message 
 . 
 user 
 ( 
 "What is the capital city of the United States?" 
 ), 
  
 Message 
 . 
 model 
 ( 
 "Washington, D.C." 
 ), 
  
 ), 
  
 samplerConfig 
  
 = 
  
 SamplerConfig 
 ( 
 topK 
  
 = 
  
 10 
 , 
  
 topP 
  
 = 
  
 0.95 
 , 
  
 temperature 
  
 = 
  
 0.8 
 ), 
 ) 
 val 
  
 conversation 
  
 = 
  
 engine 
 . 
 createConversation 
 ( 
 conversationConfig 
 ) 
 // Or with default config: 
 // val conversation = engine.createConversation() 
 // ... Use the conversation ... 
 // Close the conversation when done 
 conversation 
 . 
 close 
 ()

Conversation implements AutoCloseable , so you can use the use block for automatic resource management for one-shot or short-lived conversations:

  engine 
 . 
 createConversation 
 ( 
 conversationConfig 
 ). 
 use 
  
 { 
  
 conversation 
  
 - 
>  
 // Interact with the conversation 
 }

Sending Messages

There are three ways to send messages:

sendMessage(contents): Message : Synchronous call that blocks until the model returns a complete response. This is simpler for basic request and response interactions.
sendMessageAsync(contents, callback) : Asynchronous call for streaming responses. This is better for long-running requests or when you want to display the response as it's being generated.
sendMessageAsync(contents): Flow<Message> : Asynchronous call that returns a Kotlin Flow for streaming responses. This is the recommended approach for Coroutine users.

Synchronous Example:

  import 
  
 com.google.ai.edge.litertlm.Content 
 import 
  
 com.google.ai.edge.litertlm.Message 
 print 
 ( 
 conversation 
 . 
 sendMessage 
 ( 
 "What is the capital of France?" 
 ))

Asynchronous Example with callback:

Use sendMessageAsync to send a message to the model and receive responses through a callback.

  import 
  
 com.google.ai.edge.litertlm.Content 
 import 
  
 com.google.ai.edge.litertlm.Message 
 import 
  
 com.google.ai.edge.litertlm.MessageCallback 
 import 
  
 java.util.concurrent.CountDownLatch 
 import 
  
 java.util.concurrent.TimeUnit 
 val 
  
 callback 
  
 = 
  
 object 
  
 : 
  
 MessageCallback 
  
 { 
  
 override 
  
 fun 
  
 onMessage 
 ( 
 message 
 : 
  
 Message 
 ) 
  
 { 
  
 print 
 ( 
 message 
 ) 
  
 } 
  
 override 
  
 fun 
  
 onDone 
 () 
  
 { 
  
 // Streaming completed 
  
 } 
  
 override 
  
 fun 
  
 onError 
 ( 
 throwable 
 : 
  
 Throwable 
 ) 
  
 { 
  
 // Error during streaming 
  
 } 
 } 
 conversation 
 . 
 sendMessageAsync 
 ( 
 "What is the capital of France?" 
 , 
  
 callback 
 )

Asynchronous Example with Flow:

Use sendMessageAsync (without the callback arg) to send a message to the model and receive responses through a Kotlin Flow.

  import 
  
 com.google.ai.edge.litertlm.Content 
 import 
  
 com.google.ai.edge.litertlm.Message 
 import 
  
 kotlinx.coroutines.flow.catch 
 import 
  
 kotlinx.coroutines.launch 
 // Within a coroutine scope 
 conversation 
 . 
 sendMessageAsync 
 ( 
 "What is the capital of France?" 
 ) 
  
 . 
 catch 
  
 { 
  
 ... 
  
 } 
  
 // Error during streaming 
  
 . 
 collect 
  
 { 
  
 print 
 ( 
 it 
 . 
 toString 
 ()) 
  
 }

🔴 New: Multi-Token Prediction (MTP)

Multi-Token Prediction (MTP) is a performance optimization that significantly accelerates decode speeds. MTP is universally recommended for all tasks on GPU backends.

To use MTP, enable speculative decoding using ExperimentalFlags before initializing the engine.

  import 
  
 com.google.ai.edge.litertlm.ExperimentalApi 
 import 
  
 com.google.ai.edge.litertlm.ExperimentalFlags 
 import 
  
 com.google.ai.edge.litertlm.Backend 
 import 
  
 com.google.ai.edge.litertlm.Engine 
 import 
  
 com.google.ai.edge.litertlm.EngineConfig 
 // Enable MTP via speculative decoding 
 @OptIn 
 ( 
 ExperimentalApi 
 :: 
 class 
 ) 
 ExperimentalFlags 
 . 
 enableSpeculativeDecoding 
  
 = 
  
 true 
 val 
  
 engineConfig 
  
 = 
  
 EngineConfig 
 ( 
  
 modelPath 
  
 = 
  
 "/path/to/your/model.litertlm" 
 , 
  
 backend 
  
 = 
  
 Backend 
 . 
 GPU 
 (), 
 ) 
 val 
  
 engine 
  
 = 
  
 Engine 
 ( 
 engineConfig 
 ) 
 engine 
 . 
 initialize 
 () 
 // The same steps to create Conversation and send messages as below...

Multi-Modality

Message objects can contain different types of Content , including Text , ImageBytes , ImageFile , and AudioBytes , AudioFile .

  // Initialize the `visionBackend`, `audioBackend`, or both 
 val 
  
 engineConfig 
  
 = 
  
 EngineConfig 
 ( 
  
 modelPath 
  
 = 
  
 "/path/to/your/model.litertlm" 
 , 
  
 // Replace with your model path 
  
 backend 
  
 = 
  
 Backend 
 . 
 CPU 
 (), 
  
 // Or Backend.GPU() or Backend.NPU(...) 
  
 visionBackend 
  
 = 
  
 Backend 
 . 
 GPU 
 (), 
  
 // Or Backend.NPU(...) 
  
 audioBackend 
  
 = 
  
 Backend 
 . 
 CPU 
 (), 
  
 // Or Backend.NPU(...) 
 ) 
 // Sends a message with multi-modality. 
 // See the Content class for other variants. 
 conversation 
 . 
 sendMessage 
 ( 
 Contents 
 . 
 of 
 ( 
  
 Content 
 . 
 ImageFile 
 ( 
 "/path/to/image" 
 ), 
  
 Content 
 . 
 AudioBytes 
 ( 
 audioBytes 
 ), 
  
 // ByteArray of the audio 
  
 Content 
 . 
 Text 
 ( 
 "Describe this image and audio." 
 ), 
 ))

Defining and Using Tools

There are two ways to define tools:

With Kotlin functions (recommended for most cases)
With Open API specification (full control of the tool spec and execution)

Defining Tools with Kotlin Functions

You can define custom Kotlin functions as tools that the model can call to perform actions or fetch information.

Create a class implementing ToolSet and annotate methods with @Tool and parameters with @ToolParam .

  import 
  
 com.google.ai.edge.litertlm.Tool 
 import 
  
 com.google.ai.edge.litertlm.ToolParam 
 class 
  
 SampleToolSet 
 : 
  
 ToolSet 
  
 { 
  
 @Tool 
 ( 
 description 
  
 = 
  
 "Get the current weather for a city" 
 ) 
  
 fun 
  
 getCurrentWeather 
 ( 
  
 @ToolParam 
 ( 
 description 
  
 = 
  
 "The city name, e.g., San Francisco" 
 ) 
  
 city 
 : 
  
 String 
 , 
  
 @ToolParam 
 ( 
 description 
  
 = 
  
 "Optional country code, e.g., US" 
 ) 
  
 country 
 : 
  
 String? 
  
 = 
  
 null 
 , 
  
 @ToolParam 
 ( 
 description 
  
 = 
  
 "Temperature unit (celsius or fahrenheit). Default: celsius" 
 ) 
  
 unit 
 : 
  
 String 
  
 = 
  
 "celsius" 
  
 ): 
  
 Map<String 
 , 
  
 Any 
>  
 { 
  
 // In a real application, you would call a weather API here 
  
 return 
  
 mapOf 
 ( 
 "temperature" 
  
 to 
  
 25 
 , 
  
 "unit" 
  
 to 
  
 unit 
 , 
  
 "condition" 
  
 to 
  
 "Sunny" 
 ) 
  
 } 
  
 @Tool 
 ( 
 description 
  
 = 
  
 "Get the sum of a list of numbers." 
 ) 
  
 fun 
  
 sum 
 ( 
  
 @ToolParam 
 ( 
 description 
  
 = 
  
 "The numbers, could be floating point." 
 ) 
  
 numbers 
 : 
  
 List<Double> 
 , 
  
 ): 
  
 Double 
  
 { 
  
 return 
  
 numbers 
 . 
 sum 
 () 
  
 } 
 }

Behind the scenes, the API inspects these annotations and the function signature to generate an OpenAPI-style schema. This schema describes the tool's functionality, parameters (including their types and descriptions from @ToolParam ), and return type to the language model.

Parameter Types

The types for parameters annotated with @ToolParam can be String , Int , Boolean , Float , Double , or a List of these types (e.g., List<String> ). Use nullable types (e.g., String? ) to indicate nullable parameters. Set a default value to indicate that the parameter is optional, and mention the default value in the description in @ToolParam .

Return Type

The return type of your tool function can be any Kotlin type. The result will be converted to a JSON element before being sent back to the model.

List types are converted to JSON arrays.
Map types are converted to JSON objects.
Primitive types ( String , Number , Boolean ) are converted to the corresponding JSON primitive.
Other types are converted to strings with the toString() method.

For structured data, returning Map or a data class that will be converted to a JSON object is recommended.

Defining Tools with OpenAPI Specification

Alternatively, you can define a tool by implementing the OpenApiTool class and providing the tool's description as a JSON string conforming to the Open API specification. This method is useful if you already have an OpenAPI schema for your tool or if you need fine-grained control over the tool's definition.

  import 
  
 com.google.ai.edge.litertlm.OpenApiTool 
 class 
  
 SampleOpenApiTool 
  
 : 
  
 OpenApiTool 
  
 { 
  
 override 
  
 fun 
  
 getToolDescriptionJsonString 
 (): 
  
 String 
  
 { 
  
 return 
  
 """ 
 { 
 "name": "addition", 
 "description": "Add all numbers.", 
 "parameters": { 
 "type": "object", 
 "properties": { 
 "numbers": { 
 "type": "array", 
 "items": { 
 "type": "number" 
 } 
 }, 
 "description": "The list of numbers to sum." 
 }, 
 "required": [ 
 "numbers" 
 ] 
 } 
 } 
 """ 
 . 
 trimIndent 
 () 
  
 // Tip: trim to save tokens 
  
 } 
  
 override 
  
 fun 
  
 execute 
 ( 
 paramsJsonString 
 : 
  
 String 
 ): 
  
 String 
  
 { 
  
 // Parse paramsJsonString with your choice of parser or deserializer and 
  
 // execute the tool. 
  
 // Return the result as a JSON string 
  
 return 
  
 """{"result": 1.4142}""" 
  
 } 
 }

Registering Tools

Include instances of your tools in the ConversationConfig .

  val 
  
 conversation 
  
 = 
  
 engine 
 . 
 createConversation 
 ( 
  
 ConversationConfig 
 ( 
  
 tools 
  
 = 
  
 listOf 
 ( 
  
 tool 
 ( 
 SampleToolSet 
 ()), 
  
 tool 
 ( 
 SampleOpenApiTool 
 ()), 
  
 ), 
  
 // ... other configs 
  
 ) 
 ) 
 // Send messages that might trigger the tool 
 conversation 
 . 
 sendMessageAsync 
 ( 
 "What's the weather like in London?" 
 , 
  
 callback 
 )

The model will decide when to call the tool based on the conversation. The results from the tool execution are automatically sent back to the model to generate the final response.

Manual Tool Calling

By default, tool calls generated by the model are automatically executed by LiteRT-LM and the results from the tool execution are automatically sent back to the model to generate the next response.

If you want to manually execute tools and send results back to the model, you can set automaticToolCalling in ConversationConfig to false .

  val 
  
 conversation 
  
 = 
  
 engine 
 . 
 createConversation 
 ( 
  
 ConversationConfig 
 ( 
  
 tools 
  
 = 
  
 listOf 
 ( 
  
 tool 
 ( 
 SampleOpenApiTool 
 ()), 
  
 ), 
  
 automaticToolCalling 
  
 = 
  
 false 
 , 
  
 ) 
 )

If you disable automatic tool calling, you will need to manually execute tools and send results back to the model in your application code. The execute method of OpenApiTool will notbe called automatically when automaticToolCalling is set to false .

  // Send a message that triggers a tool call. 
 val 
  
 responseMessage 
  
 = 
  
 conversation 
 . 
 sendMessage 
 ( 
 "What's the weather like in London?" 
 ) 
 // The model returns a Message with `toolCalls` populated. 
 if 
  
 ( 
 responseMessage 
 . 
 toolCalls 
 . 
 isNotEmpty 
 ()) 
  
 { 
  
 val 
  
 toolResponses 
  
 = 
  
 mutableListOf<Content 
 . 
 ToolResponse 
> () 
  
 // There can be multiple tool calls in a single response. 
  
 for 
  
 ( 
 toolCall 
  
 in 
  
 responseMessage 
 . 
 toolCalls 
 ) 
  
 { 
  
 println 
 ( 
 "Model wants to call: 
 ${ 
 toolCall 
 . 
 name 
 } 
 with arguments: 
 ${ 
 toolCall 
 . 
 arguments 
 } 
 " 
 ) 
  
 // Execute the tool manually with your own logic. `executeTool` is just an example here. 
  
 val 
  
 toolResponseJson 
  
 = 
  
 executeTool 
 ( 
 toolCall 
 . 
 name 
 , 
  
 toolCall 
 . 
 arguments 
 ) 
  
 // Collect tool responses. 
  
 toolResponses 
 . 
 add 
 ( 
 Content 
 . 
 ToolResponse 
 ( 
 toolCall 
 . 
 name 
 , 
  
 toolResponseJson 
 )) 
  
 } 
  
 // Use Message.tool to create the tool response message. 
  
 val 
  
 toolResponseMessage 
  
 = 
  
 Message 
 . 
 tool 
 ( 
 Contents 
 . 
 of 
 ( 
 toolResponses 
 )) 
  
 // Send the tool response message to the model. 
  
 val 
  
 finalMessage 
  
 = 
  
 conversation 
 . 
 sendMessage 
 ( 
 toolResponseMessage 
 ) 
  
 println 
 ( 
 "Final answer: 
 ${ 
 finalMessage 
 . 
 text 
 } 
 " 
 ) 
  
 // e.g., "The weather in London is 25c." 
 }

Example

To try out tool use, clone the repo and run with example/ToolMain.kt :

  bazel 
 run 
 - 
 c 
 opt 
 // 
 kotlin 
 / 
 java 
 / 
 com 
 / 
 google 
 / 
 ai 
 / 
 edge 
 / 
 litertlm 
 / 
 example 
 : 
 tool 
 -- 
< abs_model_path 
>

Error Handling

API methods can throw LiteRtLmJniException for errors from the native layer or standard Kotlin exceptions like IllegalStateException for lifecycle issues. Always wrap API calls in try-catch blocks. The onError callback in MessageCallback will also report errors during asynchronous operations.