LiteRT Accelerator Test Suite (ATS)

The LiteRT Accelerator Test Suite (ATS)is a comprehensive tool used to validate the functional correctness and measure the performance of custom accelerator implementations integrated with the LiteRT framework.

Overview and Core Functionality

The primary function of ATS is to execute predefined machine learning models against a target accelerator and compare the results against the LiteRT standard CPU backend.

• Validation:The suite performs numeric validationby comparing the output tensors (activations) produced by the accelerator against those produced by the known-good CPU backend. This ensures the accelerator implementation maintains the required precision and correctness.
• Performance Metrics:It automatically captures and records critical performance details, including latencyand other relevant metrics, which are made available to the user.
• Execution:The tests are typically executed on a target device (e.g., an Android phone) and are managed by a shell scriptwrapper that handles file transfers and setup using the adb (Android Debug Bridge) tool.

Test Data (Models)

The ATS suite uses a collection of widely-used .tflite modelsas its test data. Input data is generated randomly based on the data type and can be seeded as needed.

Included Models

The following models are automatically included and downloaded for testing (subject to change):

• hf_all_minilm_l6_v2
• hf_mobilevit_small
• qai_hub_midas
• qai_hub_real_esrgan_x4plus
• torchvision_mobilenet_v2
• torchvision_resnet18
• torchvision_squeezenet1_1
• u2net_lite
• whisper_tiny_decoder
• whisper_tiny_encoder
• yamnet
• yolo11n

Manual Model Retrieval

While models are automatically downloaded during a bazel run , you can manually retrieve the entire model set using wget :

 wget  
-p  
-O  
<target_file>  
https://storage.googleapis.com/litert/ats_models.tar.gz 

Defining an ATS Suite with Bazel

Use the litert_define_ats Bazel macro to configure and define an ATS testing target specific to their accelerator.

The macro automatically creates tworunnable targets:

1. The standard on-device JIT test(for execution and validation).
2. A dedicated AOT "compile only" mode test(for host compilation).

Example litert_define_ats Usage

The example defines an ATS suite named example_ats for an accelerator with the backend name example :

  # Emits aot-mode and jit-mode test targets, one for running compilation test on host 
 # and another for running JIT and inference on device 
 # These targets are named with their respective suffix attribute. 
 litert_define_ats 
 ( 
 name 
 = 
 "example_ats" 
 , 
 backend 
 = 
 "example" 
 , 
 compile_only_suffix 
 = 
 "_aot" 
 , 
 do_register 
 = 
 [ 
 "*mobilenet*" 
 , 
 ], 
 extra_flags 
 = 
 [ 
 "--limit=1" 
 ], 
 jit_suffix 
 = 
 "" 
 , 
 ) 
 

Execution

To execute the standard test targeted for android (which handles all adb operations):

  # Handles environment setup, and build + push of library and data dependencies to the device, 
 # executes the suite on the target. 
bazel  
run  
-c  
opt  
--config = 
android_arm64  
:example_ats 

To execute the AOT compilation test:

  # Handle environment setup, and builds library dependencies for host platform. 
 # Executes the ats compile only flow. The "--compile_mode" flag is already 
 # bound to the program arguments. 
bazel  
run  
:example_ats_aot 

Linux Execution (Host)

For Linux execution, where ATS is being run on the same machine doing the build, users will need to use the :ats binary directly:

 bazel  
run  
-c  
opt  
:ats 

IoT Execution

For IoT execution, users will need to build the binary on the host and manually push it to their device.

Command-Line Flags

The ats executable accepts several flags for granular control over testing and reporting.

Using the litert_device_script build utilities for ATS

The ATS targets users execute automatically include a shell entry point which handles all of the environment setup, and any pushing of required libraries when the target device differs from the host on which the build was completed (e.g. adb push ).

This functionality is provided generically through the litert_device_script utilities which ATS builds use under the hood. There is a registration process accelerators must do to access this build functionality. In addition to supporting ats , these utilities can be used in a standalone fashion to simulate cc_binary and cc_test meant to be executed on a device different from the build host requiring pushed dependencies.

Backend Registration

To enable a new accelerator for use with litert_device_script (and therefore ATS), its required libraries must be registered in the litert_device_common.bzl Bazel file. Registration is based on a unique "backend" namewhich maps to a set of buildable or pre-compiled libraries needed for LiteRT to operate with that accelerator.

Registration Steps

1. Define a BackendSpec function:Create a function that returns a dictionary containing your new accelerator's specification.

2. Specify Libraries ( libs ):This is a list of tuples detailing the Bazel target path for the shared library and the environment variable ( LD_LIBRARY_PATH ) required for the device linker to find it.

   • Dispatch Library:Required for runtime execution.
   • Compiler Plugin Library:Required for AOT compilation mode.

3. Specify Library Names ( plugin , dispatch ):Provide the file names of the plugin and dispatch libraries.

4. Register the Spec:Merge your new spec function into the main _Specs function to make it available by its unique backend ID.

Example Registration ( _ExampleSpec )

The following code from litert_device_common.bzl illustrates how the "example" accelerator is registered:

  def 
  
 _ExampleSpec 
 (): 
 return 
 { 
 # The unique backend ID 
 "example" 
 : 
 BackendSpec 
 ( 
 id 
 = 
 "example" 
 , 
 libs 
 = 
 [ 
 # Dispatch Library and how to find it on device 
 ( 
 "//third_party/odml/litert/litert/vendors/examples:libLiteRtDispatch_Example.so" 
 , 
 "LD_LIBRARY_PATH" 
 ), 
 # Compiler Plugin Library 
 ( 
 "//third_party/odml/litert/litert/vendors/examples:libLiteRtCompilerPlugin_Example.so" 
 , 
 "LD_LIBRARY_PATH" 
 ), 
 ], 
 plugin 
 = 
 "libLiteRtCompilerPlugin_Example.so" 
 , 
 dispatch 
 = 
 "libLiteRtDispatch_Example.so" 
 , 
 ), 
 } 
 # ... (Other specs are defined here) 
 def 
  
 _Specs 
 ( 
 name 
 ): 
 # Your new spec function must be included here 
 return 
 ( 
 _QualcommSpec 
 () 
 | 
 _GoogleTensorSpec 
 () 
 | 
 _MediatekSpec 
 () 
 | 
 _CpuSpec 
 () 
 | 
 _GpuSpec 
 () 
 | 
 _ExampleSpec 
 ())[ 
 name 
 ] 
 

Leveraging Registration with litert_device_exec

Once registered, use the litert_device_exec and related macros with the new backend_id . This macro automatically bundles the required libraries and any specified data files with the target binary.

  cc_binary 
 ( 
 name 
 = 
 "example_bin" 
 , 
 srcs 
 = 
 [ 
 "example_bin.cc" 
 ], 
 ) 
 litert_device_exec 
 ( 
 name 
 = 
 "example_bin_device" 
 , 
 backend_id 
 = 
 "example" 
 , 
 # Uses the libraries registered under "example" 
 data 
 = 
 [ 
 "//third_party/odml/litert/litert/test:testdata/constant_output_tensor.tflite" 
 , 
 ], 
 target 
 = 
 ":example_bin" 
 , 
 ) 
 

Running this target ( bazel run ... :example_bin_device ) will:

1. Build the example_bin C++ binary.
2. Push the binary, libLiteRtDispatch_Example.so , libLiteRtCompilerPlugin_Example.so , and the .tflite file to the device.
3. Execute the binary using adb shell .

Note on Device Paths:The canonical location for files on the device mirrors Bazel's runfile tree, specifically /data/local/tmp/runfiles/runfiles_relative_path . The device script automatically handles setting the appropriate paths for the dynamic linker.

Compilation Mode (AOT)

For accelerators that support an Ahead-of-Time (AOT) compilationstep, ATS can be executed in a dedicated "compile mode".

• Purpose:This mode is designed to be run on a workstation(host machine), not the target device. It compiles the models for the specified target hardware without executing them.
• Output:All compiled models are output to a designated directory on the workstation.
• Activation:The ATS build macros will emit a specific target for aot where libraries are built for the host platform. This flow can be enabled on any binary with the --compile_mode flag, but is is automatically bound to the arguments of the aot build.

Future Expansion

The suite is planned to be expanded to include dedicated tests for single operations (ops), in addition to full models.