liveBook · Manning

Chapter 11: Targeting Edge and Mobile Devices

What you will learn in chapter 11 of Quantization and Fast Inference

The introduction explains why edge deployment must account for sustained latency, energy use, and operator placement across a device’s compute blocks.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains why edge deployment must account for sustained latency, energy use, and operator placement across a device’s compute blocks.

Key terms: joules per inference, thermal knee, sustained workload, latency, system-on-chip (SoC), CPU, GPU, dedicated ML accelerator, operator, fallback

11.1 Set latency and power budgets

This section explains how to filter edge-device configurations by user-visible latency, sustained thermal performance, and power before tuning them.

Key terms: latency-and-power budget, user-visible path, single-shot latency, sustained throughput retention, thermal budget, mean power, cold-start latency, energy per inference, first-token latency, prompt length

11.1.1 The latency × power design space

This section maps edge workloads by latency to first useful output and steady-state power, then uses M3 and Pixel measurements to show how workload, format, and prompt length affect those measures.

Key terms: latency to first useful output, p50, time-to-first-token (TTFT), mean power, steady-state window, vision, audio, LLM TTFT, edge stack, power cluster

11.1.2 Sustained throughput and the thermal knee

This section explains why edge-device performance must be measured under sustained inference, how to compare thermal effects across workloads, and how benchmark duration changes the result.

Key terms: thermal knee, throttling penalty, junction-temperature loop, virtual-skin-temperature loop, throughput retention, samples/s, encoder-segments/s, tokens/s, M3 ANE thermal knee, M3 audio warm-up

11.1.3 The trust ladder for power numbers

The section explains how to judge device power numbers by their measurement source and how to handle an iPhone power value that cannot be measured in milliwatts.

Key terms: powermetrics, IOReport, BatteryManager, fuel-gauge IC, trust ladder, MetricKit, Energy Impact, power_mw, null

11.1.4 The budget filter, applied

Applies latency, sustained-performance, and power budgets to measured edge-device results, showing why feasible configurations and limiting factors differ by workload.

Key terms: budget filter, feasibility envelope, p95 latency, mean power, sustained-throughput retention, latency to first useful output, per-token decode rate, per-prefill-call latency, TTFT, energy per inference

11.2 Android with TFLite and LiteRT-LM

Introduces Android’s two LiteRT runtimes and explains why TFLite conversion choices must be evaluated alongside activation types and device delegates.

Key terms: TensorFlow Lite, LiteRT, LiteRT-LM, .tflite, .litertlm, delegate, fp32, dynrange, int8, int16x8

11.2.1 Four paths, four graphs

Compares four TFLite conversion paths by checking their actual tensor dtypes and interpreting their size, accuracy, and latency trade-offs.

Key terms: per-tensor dtype map, .tflite flatbuffer, get_tensor_details(), top-1, top-5, calibration mis-bracketing, dynamic-range, INT8 weight tensors, FP32 activation tensors, per-op dequant

11.2.2 The Pixel runtime matrix

The section uses a Pixel 10 Pro runtime matrix to show why quantization variant, delegate, workload size, and actual backend dispatch must be evaluated together.

Key terms: variant × delegate, XNNPACK, NNAPI, empty cells, dynamic-range, thread-pool wake-up cost, INT16×8, reference interpreter, GPU dispatch tax, matmul-cost threshold

11.2.3 LiteRT-LM as TFLite's transformer sibling

This section contrasts TFLite and LiteRT-LM artifacts and runtime controls, then explains why transformer inference needs separate prefill and decode measurements.

Key terms: .tflite, .litertlm, Interpreter, tokenizer config, KV-cache layout, sampling configuration default, delegate, Backend hint, EngineConfig, compute_units

11.3 Apple silicon with Core ML and MLX

This section distinguishes Core ML and MLX deployment roles and explains how to interpret the Apple-device measurements used in the chapter.

Key terms: Core ML, .mlpackage, MLX, prefill, autoregressive-decode, MLComputeUnits, compute-unit hint, per-op fallback, Apple Neural Engine, op-support table

11.3.1 The compute-unit dispatch knob

This section explains how Core ML compute-unit settings govern eligible hardware and why dispatch matters more than weight variant for the measured EfficientNet-Lite0 workload.

Key terms: MLComputeUnits, cpuOnly, cpuAndGPU, cpuAndNeuralEngine, all, compute-unit hint, weight variant, INT8 weight quantization, ANE, Xcode Performance Reports

11.3.2 The ANE op-support cliff

This section explains why unsupported transformer operators keep iPhone Llama models off the Apple Neural Engine, how to verify their actual routing, and what the Core ML Llama API provides.

Key terms: Neural Engine, supported-op table, compute-unit hint, GPU fallback, Xcode Performance Report, MLComputePlan, Llama, ANE coverage, INT4 per-block, INT8 per-block

11.3.3 MLX as Core ML's transformer sibling

This section contrasts Apple silicon runtime paths for vision and transformer workloads, then explains how weight quantization affects MLX decode throughput.

Key terms: Core ML, PyTorch MPS, MLX, Metal, Neural Engine, compute-unit hint, autoregressive decode, coremltools.convert(...), .mlpackage, memory-bandwidth-bound

11.3.4 Sustained throughput and the palettize cliff

This section shows why sustained throughput, weight format, and actual operator routing must be measured rather than inferred from short runs, file size, or device settings.

Key terms: steady-state, soft-throttle, last-30-second throughput, memory-bandwidth-bound, palettize_weights, palette, lookup-table decompression, variant × device sweep, weight variant, MLComputeUnits

11.4 Phone, SBC, and small-box tiers

This section explains how deployment surface determines the phone, SBC, or small-box tier, and how each tier changes runtime choices, accelerator support, and validation work.

Key terms: deployment surface, phone tier, SBC tier, small-box tier, LiteRT on Linux, libtensorflowlite.so, XNNPACK, .tflite flatbuffer, NNAPI, Edge TPU compiler

11.5 Pre and post-processing

Shows why edge-device latency must include pre- and post-processing, and how platform-native implementations can change which stage is worth optimizing.

Key terms: inference-only, end-to-end p95, pre/post, SLA, log-mel, Kotlin radix-2, PIL, OpenCV, vImage, platform-native primitives

11.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 10: Running on CPUs and Using GGUF Chapter 12: Delivering Proven Results with a Four-Bit LLM Server and an On-Device Vision Pipeline →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.