chapter eleven

11 Targeting Edge and Mobile Devices

 

This chapter covers

  • Setting achievable latency and power budgets
  • Driving TensorFlow Lite's integer paths
  • Leveraging NNAPI and Core ML backends
  • Selecting among phone, SBC, and small-box deployment tiers
  • Instrumenting pre and post-processing

The edge regime swaps both axes that drove a data-center deployment, where no inference ever had to ask whether it would overheat the room or drain a battery. The cost dial stops being dollars per million tokens and becomes joules per inference. The failure mode stops being a kernel that silently dispatches to FP32 and becomes a thermal knee five minutes into a sustained workload, where the latency you measured in the first thirty seconds is no longer the latency a user sees.

This lesson is for the ML engineer who has shipped quantized models to a server fleet but has never owned a build that runs on a phone.

11.1 Set latency and power budgets

11.1.1 The latency × power design space

11.1.2 Sustained throughput and the thermal knee

11.1.3 The trust ladder for power numbers

11.1.4 The budget filter, applied

11.2 Android with TFLite and LiteRT-LM

11.2.1 Four paths, four graphs

11.2.2 The Pixel runtime matrix

11.2.3 LiteRT-LM as TFLite's transformer sibling

11.3 Apple silicon with Core ML and MLX

11.3.1 The compute-unit dispatch knob

11.3.2 The ANE op-support cliff

11.3.3 MLX as Core ML's transformer sibling

11.3.4 Sustained throughput and the palettize cliff

11.4 Phone, SBC, and small-box tiers

11.5 Pre and post-processing

11.6 Summary