liveBook · Manning

LiteRT-LM

As explained in Quantization and Fast Inference

LiteRT-LM is a runtime for deploying and executing transformer models, with bundle-based inputs and streaming generation APIs.

LiteRT-LM runs transformer models on Android using .litertlm bundles and a streaming generation API. Unlike graph-inference runtimes, it packages transformer-specific metadata with the model and exposes prefill and per-token decode as distinct phases. section 11.2 section 11.2.3

The runtime’s interface reflects transformer deployment needs:

  • A bundle includes tokenizer configuration, KV-cache layout, and default sampling configuration. section 11.2.3
  • App code calls sendMessageAsync and receives token deltas as a Kotlin Flow. section 11.2.3
  • EngineConfig accepts a CPU or GPU backend hint, but the runtime may route work differently. section 11.2.3

Android deployment

Llama-3.2-1B-Instruct uses LiteRT-LM bundles. On the Tensor G5, INT8 transformer workloads route to the GPU regardless of the backend hint; record compute_units to identify where execution ran. section 11.2 section 11.2.3

How Quantization and Fast Inference covers it

How deep it goes
Details bundle metadata, streaming calls, backend hints, and separate prefill and decode measurements.
Examples
Uses Llama-3.2-1B-Instruct and Tensor G5 GPU routing to illustrate deployment behavior.
What you can do after
Apply LiteRT-LM by supplying a bundle, configuring a backend hint, and measuring prefill and decode separately.
How it connects
Connects transformer deployment to TFLite, flatbuffers, tokenization, and KV caching.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.