LiteRT-LM runs transformer models on Android using .litertlm bundles and a streaming generation API. Unlike graph-inference runtimes, it packages transformer-specific metadata with the model and exposes prefill and per-token decode as distinct phases. section 11.2 section 11.2.3
The runtime’s interface reflects transformer deployment needs:
- A bundle includes tokenizer configuration, KV-cache layout, and default sampling configuration. section 11.2.3
- App code calls sendMessageAsync and receives token deltas as a Kotlin Flow. section 11.2.3
- EngineConfig accepts a CPU or GPU backend hint, but the runtime may route work differently. section 11.2.3
Android deployment
Llama-3.2-1B-Instruct uses LiteRT-LM bundles. On the Tensor G5, INT8 transformer workloads route to the GPU regardless of the backend hint; record compute_units to identify where execution ran. section 11.2 section 11.2.3
