liveBook · Manning

Delegate

As explained in Quantization and Fast Inference

A delegate is a runtime component that executes supported model operations through a particular hardware backend or optimized operator library.

A TFLite delegate binds an interpreter to an execution backend. Attach it when constructing the interpreter; the runtime either accepts the model graph or rejects it at compile time. Delegate compatibility depends on the model’s quantization variant, so choose the two together. section 11.2.3 section 11.2.2

On the Pixel 10 Pro, delegate behavior varies by variant:

  • XNNPACK has kernels for every tested TFLite variant. section 11.2.2
  • NNAPI accepts FP32 and dynamic-range vision models, but rejects INT8 and INT16×8 at compile time under Android 16. section 11.2.2
  • INT16×8 takes about 143 ms across accepting delegates because the supported-ops set falls back to the reference interpreter. section 11.2.2

LiteRT-LM uses a different interface: its CPU or GPU backend setting is a hint, not a TFLite delegate binding. On the Tensor G5, INT8 transformer workloads route to the GPU regardless of the hint. section 11.2.3

Figure 11.7 Pixel 10 Pro runtime matrix across vision (EfficientNet TFLite), audio (Whisper-tiny TFLite encoder), and language (Llama-3.2-1B LiteRT-LM), shared log y-axis. Vision panel shows four TFLite variants × four delegates (XNNPACK one- and four-thread, GPU over the Tensor G5's Imagination PowerVR DXT-48-1536, NNAPI). NNAPI rejects INT8 and INT16×8 at compile time and the cells appear empty. INT16×8 is a uniform ~143 ms across delegates because XNNPACK has no fused kernels for the supported-ops set and every cell falls back to the reference interpreter. Audio panel shows two TFLite variants (FP32 and dynamic-range, since Whisper-tiny is not calibrated on this fleet because no representative mel-spectrogram dataset is in scope) × three delegates. The XNNPACK 1T column is absent because the AWS Device Farm sweep collected only XNNPACK 4T / NNAPI / GPU for the audio matrix. LLM panel shows LiteRT-LM's single INT8 bundle, three prompt lengths (32 / 256 / 1024 tokens), TTFT only. LiteRT-LM does not expose a variant axis at the application API surface, every cell ran on the GPU.

Compare tested variant–delegate combinations across vision, audio, and language workloads, including combinations the runtime rejects. section 11.2.2

How Quantization and Fast Inference covers it

How deep it goes
Traces interpreter attachment, compile-time graph rejection, and variant-specific compatibility, but not delegate implementation internals.
Examples
Compares XNNPACK and NNAPI across EfficientNet vision variants on a Pixel 10 Pro.
What you can do after
Evaluate delegate compatibility against model variants and recognize compile-time rejection or fallback costs.
How it connects
Connects TFLite delegates with quantization variants and contrasts them with LiteRT-LM backend hints.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.