Chapter 11: Targeting Edge and Mobile Devices
The introduction explains why edge deployment must account for sustained latency, energy use, and operator placement across a device’s compute blocks.
After this chapter you will be able to
- Sustained edge performance budgets. Explain why edge performance must be judged by energy per inference and latency under sustained use, rather than by server cost or a short initial latency measurement.
- SoC operator placement and fallback. Describe the phone’s system-on-chip as containing CPU, GPU, and dedicated ML accelerator blocks, and explain what happens when an operator cannot run on the accelerator.
- Edge deployment budget filter. Explain why latency and power budgets filter configurations before optimization, and why acceptable latency depends on the user-visible task rather than a single universal target.
- Sustained power and device tradeoffs. Explain why a cold-start latency result cannot establish that a configuration will work on a phone, using the M3 and Pixel comparison to illustrate the speed–power tradeoff.
- Workload-dependent binding constraints. Describe how a change in workload can change which budget constraint matters most, using the Whisper-tiny and prompt-length examples.
- Latency and power axes. Explain what each axis of the design-space figure measures and why the latency measure represents the user's first response.
- Workload driven latency differences. Explain what the M3 and Pixel comparisons suggest about the dominant source of latency differences across edge workloads.
- M3 format power tradeoff. Describe why the M3 Core ML variants form a power cluster rather than a latency cluster, and what the palettized results show about model size and power.
- Thermal knee and throttling. Explain why cold-start latency can differ from sustained performance and how mobile temperature controls can cause a thermal knee.
- Cross-modality throughput retention. Describe how throughput retention is calculated and why it lets the figure compare vision, audio, and LLM workloads.
- Sustained benchmark window selection. Justify the section's benchmark window and explain when a longer measurement is needed.
- Power measurement trust hierarchy. Distinguish the sources and limits of Mac and Pixel power readings, and explain why a cross-device power figure must disclose how each number was obtained.
- Unmeasurable iPhone power. Explain why the available iPhone tools cannot supply a milliwatt reading for a short inference window, and describe how the missing value should appear in records and power-axis figures.
- Budget Feasibility Before Optimization. Explain how to filter measured configurations against an edge budget before choosing the fastest one, using the 33 ms p95 and 5 W vision envelope as an example.
- Workload Specific Budget Bottlenecks. Explain which budget constraint becomes decisive for vision, speech recognition, and LLM inference, using the section's measured examples.
- Android runtime roles and interfaces. Distinguish TFLite/LiteRT’s general-purpose op-graph path from LiteRT-LM’s transformer path, including the artifacts and app-facing choices each uses.
- TFLite paths and device binding. Compare the four TFLite converter paths by weight and activation type, and explain why the target-device delegate is a separate consideration rather than another point on a quantization scale.
- Per-tensor dtype verification. Explain how to inspect a TFLite flatbuffer's per-tensor dtype map and why a successful conversion alone does not establish what was quantized.
- Top-one versus top-five diagnosis. Explain why the measured top-1 and top-5 results point to calibration mis-bracketing rather than a general loss of candidate classes.
- Dynamic-range execution trade-off. Describe what the dynamic-range path quantizes and why it preserves accuracy and reduces size without matching INT8 latency.
- Fully integer delegate eligibility. Explain why the INT8 graph qualifies for integer hardware delegates and why its observed accuracy cliff is attributed to calibration data.
- Variant and delegate compatibility. Explain why a TFLite quantization variant and delegate must be evaluated as a pair, using the Pixel runtime matrix to distinguish supported combinations from rejected ones.
- Workload-dependent GPU dispatch cost. Explain why the GPU is much slower for the small EfficientNet inference but comparatively less costly for larger work, and state the resulting workload-dependent rule for choosing it.
- LiteRT-LM bundle and backend. Distinguish LiteRT-LM's bundle and backend-hint choices from TFLite's variant and delegate choices, and explain why measured results must report the backend that actually ran.
- Transformer bundle and streaming interface. Describe how a LiteRT-LM bundle and its app interface differ from a TFLite file and Interpreter. Explain why the bundle includes tokenizer, KV-cache, and sampling metadata.
- Delegate binding versus backend hint. Distinguish TFLite delegate acceptance from LiteRT-LM backend selection, and explain why measured compute units must be recorded.
- Prefill decode and response latency. Explain what prefill and decode do, how their costs respond to prompt length, and why TTFT is the user-facing response-latency measure here.
- Core ML and MLX roles. Distinguish Core ML’s broad Apple-device coverage from MLX’s M-series-only transformer decode role, including Core ML’s use for vision, audio, and LLM prefill.
- Core ML compute-unit hint. Explain that Core ML’s MLComputeUnits setting is a hint that limits eligible compute units, not a guarantee that every operation runs on the Apple Neural Engine.
- Core ML compute-unit choices. Explain what each MLComputeUnits setting permits and why a compute-unit hint does not necessarily dictate where an eligible operation runs.
- Dispatch versus weight variant. Explain what the EfficientNet-Lite0 latency comparison shows about prioritizing compute-unit dispatch over weight-variant choice, including the different CPU, GPU, and ANE patterns.
- Operator support and silent fallback. Explain how Neural Engine operator support determines execution routing and how to check where a model actually runs.
- Llama quantization and ANE coverage. Explain why changing the tested iPhone Llama weight formats does not move its execution onto the Neural Engine.
- Apple runtime accelerator access. Distinguish which accelerators Core ML, PyTorch MPS, and MLX can reach, and explain why routing a vision model through Metal rather than the Neural Engine incurs a large latency cost.
- Quantized decode bandwidth limits. Explain why four-bit MLX weights improve short-prompt decode throughput and why that advantage narrows with a longer prompt.
- Workload dependent sustained throughput. Explain why a thermally representative late-window throughput measurement is more useful than an initial burst for a continuously running workload, and why throughput can fall, rise, or stay flat over the same window.
- Palettized weight latency cliff. Explain why palettized weights made the small Whisper-tiny encoder slower on the Neural Engine despite a smaller weight file, and describe how to check whether that tradeoff applies to another model.
- Deployment Surface Tier Selection. Explain why the deployment surface, rather than the model, determines the tier. Describe when a consumer phone, a fixed installation, or a throughput-bound 24/7 workload calls for the phone, SBC, or small-box tier.
- Attached Accelerator Op Support. Explain how an attached Coral or Hailo accelerator changes an SBC deployment and why INT8 conversion alone does not guarantee accelerator execution.
- Small-Box Vendor Runtime Lock-In. Describe how a small-box appliance differs from the TFLite-based tiers and why targeting multiple vendors requires separate conversion and validation pipelines.
- Cross-Runtime Calibration Drift. Explain why the same source model and representative dataset can yield different INT8 accuracy across small-box vendors, and state the resulting calibration obligation.
- End-to-end latency accounting. Explain why inference-only timing is useful for comparing runtimes but insufficient for a user-facing SLA. State that the SLA should report end-to-end p95, with inference-only latency alongside it as a diagnostic.
- Platform-dependent pipeline bottlenecks. Describe how the measured audio and vision preprocessing costs change optimization priorities across devices. Use the Pixel log-mel and M3 Python-versus-vImage vision examples to explain why inference-only rankings can mislead.
- Native preprocessing remedies. Explain the section’s preferred ways to reduce pre- and post-processing latency: use platform-native primitives when available, and for a missing primitive such as Android log-mel, put the operations in the model graph or bind a tuned native library.
What it builds on
- Build a calibration set that actually represents production (chapter 4) · Representative data for calibration.
- Quantize transformer key-value cache with mixed precision (chapter 3) · The role of the key-value cache in transformer inference.
- Build TensorRT engines with real integer execution (chapter 9) · The TensorRT toolchain used for Jetson.
- Deploy on Intel CPU with OpenVINO and NNCF (chapter 9) · The OpenVINO and NNCF deployment path used for Intel.
Section by section
Introduction
The introduction explains why edge deployment must account for sustained latency, energy use, and operator placement across a device’s compute blocks.
- Sustained edge performance budgets: Explain why edge performance must be judged by energy per inference and latency under sustained use, rather than by server cost or a short initial latency measurement.
Why it matters: A fast first run may not meet a user’s latency or battery expectations during continued use. - SoC operator placement and fallback: Describe the phone’s system-on-chip as containing CPU, GPU, and dedicated ML accelerator blocks, and explain what happens when an operator cannot run on the accelerator.
Why it matters: Choosing an accelerator path does not guarantee that every operator runs there.
Key terms: joules per inference, thermal knee, sustained workload, latency, system-on-chip (SoC), CPU, GPU, dedicated ML accelerator, operator, fallback
11.1 Set latency and power budgets
This section explains how to filter edge-device configurations by user-visible latency, sustained thermal performance, and power before tuning them.
- Edge deployment budget filter: Explain why latency and power budgets filter configurations before optimization, and why acceptable latency depends on the user-visible task rather than a single universal target.
Why it matters: A fast configuration is useful only if it fits the latency and power limits of its deployment. - Sustained power and device tradeoffs: Explain why a cold-start latency result cannot establish that a configuration will work on a phone, using the M3 and Pixel comparison to illustrate the speed–power tradeoff.
Why it matters: Checking sustained behavior prevents a configuration that looks fast in a brief test from slowing down in users' hands. - Workload-dependent binding constraints: Describe how a change in workload can change which budget constraint matters most, using the Whisper-tiny and prompt-length examples.
Why it matters: Measuring the wrong constraint can hide the reason a configuration fails for a particular workload.
Key terms: latency-and-power budget, user-visible path, single-shot latency, sustained throughput retention, thermal budget, mean power, cold-start latency, energy per inference, first-token latency, prompt length
11.1.1 The latency × power design space
This section maps edge workloads by latency to first useful output and steady-state power, then uses M3 and Pixel measurements to show how workload, format, and prompt length affect those measures.
- Latency and power axes: Explain what each axis of the design-space figure measures and why the latency measure represents the user's first response.
Why it matters: These two measures let you compare changes in responsiveness and power without treating them as the same outcome. - Workload driven latency differences: Explain what the M3 and Pixel comparisons suggest about the dominant source of latency differences across edge workloads.
Why it matters: Choosing a workload matters more to the latency comparison than simply naming the device. - M3 format power tradeoff: Describe why the M3 Core ML variants form a power cluster rather than a latency cluster, and what the palettized results show about model size and power.
Why it matters: A smaller weight representation may improve neither response time nor power on the target hardware. - Palettization weight representation: Describe how Core ML palettization represents weights and distinguish its bit count from the precision used for multiplication.
Why it matters: The representation explains why a 4-bit palettized file is not the same thing as 4-bit multiplication. - Pixel LLM prompt scaling: Describe how prompt length changes the measured Pixel LLM first-token latency and power.
Why it matters: A longer prompt can worsen first-token responsiveness without producing a corresponding rise in measured power.
Key terms: latency to first useful output, p50, time-to-first-token (TTFT), mean power, steady-state window, vision, audio, LLM TTFT, edge stack, power cluster
11.1.2 Sustained throughput and the thermal knee
This section explains why edge-device performance must be measured under sustained inference, how to compare thermal effects across workloads, and how benchmark duration changes the result.
- Thermal knee and throttling: Explain why cold-start latency can differ from sustained performance and how mobile temperature controls can cause a thermal knee.
Why it matters: A fast cold run may promise performance the device cannot maintain. - Cross-modality throughput retention: Describe how throughput retention is calculated and why it lets the figure compare vision, audio, and LLM workloads.
Why it matters: Retention shows whether performance lasts without pretending that samples, audio segments, and tokens are the same unit. - Contrasting device throughput patterns: Distinguish the M3 vision slowdown and audio warm-up from the Pixel results, without treating one device's pattern as universal.
Why it matters: Sustained throughput can fall, rise, or stay flat depending on the device and workload. - Sustained benchmark window selection: Justify the section's benchmark window and explain when a longer measurement is needed.
Why it matters: A benchmark that ends too soon can miss either delayed throttling or delayed speedup.
Key terms: thermal knee, throttling penalty, junction-temperature loop, virtual-skin-temperature loop, throughput retention, samples/s, encoder-segments/s, tokens/s, M3 ANE thermal knee, M3 audio warm-up
11.1.3 The trust ladder for power numbers
The section explains how to judge device power numbers by their measurement source and how to handle an iPhone power value that cannot be measured in milliwatts.
- Power measurement trust hierarchy: Distinguish the sources and limits of Mac and Pixel power readings, and explain why a cross-device power figure must disclose how each number was obtained.
Why it matters: A power figure is useful only when readers can judge the quality of the readings behind it. - Unmeasurable iPhone power: Explain why the available iPhone tools cannot supply a milliwatt reading for a short inference window, and describe how the missing value should appear in records and power-axis figures.
Why it matters: Marking unavailable power as missing prevents a chart from implying a measurement that was never made.
Key terms: powermetrics, IOReport, BatteryManager, fuel-gauge IC, trust ladder, MetricKit, Energy Impact, power_mw, null
11.1.4 The budget filter, applied
Applies latency, sustained-performance, and power budgets to measured edge-device results, showing why feasible configurations and limiting factors differ by workload.
- Budget Feasibility Before Optimization: Explain how to filter measured configurations against an edge budget before choosing the fastest one, using the 33 ms p95 and 5 W vision envelope as an example.
Why it matters: The fastest measured option may violate the power limit of the device you intend to ship. - Cross Device Measurement Comparability: Distinguish the latency measurements in the cross-device chart and explain why apparent device rankings can change with modality and model variant.
Why it matters: Treating different latency measures as interchangeable can produce a misleading device comparison. - Workload Specific Budget Bottlenecks: Explain which budget constraint becomes decisive for vision, speech recognition, and LLM inference, using the section's measured examples.
Why it matters: A budget that works for one workload may miss the cost that dominates another on the same class of device.
Key terms: budget filter, feasibility envelope, p95 latency, mean power, sustained-throughput retention, latency to first useful output, per-token decode rate, per-prefill-call latency, TTFT, energy per inference
11.2 Android with TFLite and LiteRT-LM
Introduces Android’s two LiteRT runtimes and explains why TFLite conversion choices must be evaluated alongside activation types and device delegates.
- Android runtime roles and interfaces: Distinguish TFLite/LiteRT’s general-purpose op-graph path from LiteRT-LM’s transformer path, including the artifacts and app-facing choices each uses.
Why it matters: Choosing an Android runtime also determines the artifact and deployment choices the app must handle. - TFLite paths and device binding: Compare the four TFLite converter paths by weight and activation type, and explain why the target-device delegate is a separate consideration rather than another point on a quantization scale.
Why it matters: A model’s converter preset alone does not tell you what execution path will work on a phone.
Key terms: TensorFlow Lite, LiteRT, LiteRT-LM, .tflite, .litertlm, delegate, fp32, dynrange, int8, int16x8
11.2.1 Four paths, four graphs
Compares four TFLite conversion paths by checking their actual tensor dtypes and interpreting their size, accuracy, and latency trade-offs.
- Per-tensor dtype verification: Explain how to inspect a TFLite flatbuffer's per-tensor dtype map and why a successful conversion alone does not establish what was quantized.
Why it matters: Checking the produced graph prevents a conversion flag from being mistaken for proof of integer execution. - Top-one versus top-five diagnosis: Explain why the measured top-1 and top-5 results point to calibration mis-bracketing rather than a general loss of candidate classes.
Why it matters: The two accuracy measures help identify a calibration problem instead of blaming integer computation itself. - Dynamic-range execution trade-off: Describe what the dynamic-range path quantizes and why it preserves accuracy and reduces size without matching INT8 latency.
Why it matters: It is a useful size-versus-accuracy choice when a calibration dataset cannot be shipped, but it is not the fastest integer path. - Fully integer delegate eligibility: Explain why the INT8 graph qualifies for integer hardware delegates and why its observed accuracy cliff is attributed to calibration data.
Why it matters: A fully integer graph enables delegate dispatch, while representative calibration remains necessary for accuracy. - INT16×8 accuracy latency trade-off: Describe what wider INT16 activations can recover and why the measured INT16×8 path is unsuitable for phone deployment on this model.
Why it matters: A small accuracy gain does not justify this path's latency cost for a phone on the measured model.
Key terms: per-tensor dtype map, .tflite flatbuffer, get_tensor_details(), top-1, top-5, calibration mis-bracketing, dynamic-range, INT8 weight tensors, FP32 activation tensors, per-op dequant
11.2.2 The Pixel runtime matrix
The section uses a Pixel 10 Pro runtime matrix to show why quantization variant, delegate, workload size, and actual backend dispatch must be evaluated together.
- Variant and delegate compatibility: Explain why a TFLite quantization variant and delegate must be evaluated as a pair, using the Pixel runtime matrix to distinguish supported combinations from rejected ones.
Why it matters: A quantized model is not a viable phone configuration unless its intended runtime accepts the graph. - Threading and emulation limits: Describe why four threads materially help only the tested dynamic-range vision path, and why INT16×8 vision latency remains similar across delegates that accept it.
Why it matters: More threads or a different delegate does not automatically accelerate a quantized path. - Workload-dependent GPU dispatch cost: Explain why the GPU is much slower for the small EfficientNet inference but comparatively less costly for larger work, and state the resulting workload-dependent rule for choosing it.
Why it matters: The fastest delegate for one model need not be the fastest for another on the same phone. - LiteRT-LM bundle and backend: Distinguish LiteRT-LM's bundle and backend-hint choices from TFLite's variant and delegate choices, and explain why measured results must report the backend that actually ran.
Why it matters: Reporting a requested backend instead of the executed one can misrepresent an edge benchmark.
Key terms: variant × delegate, XNNPACK, NNAPI, empty cells, dynamic-range, thread-pool wake-up cost, INT16×8, reference interpreter, GPU dispatch tax, matmul-cost threshold
11.2.3 LiteRT-LM as TFLite's transformer sibling
This section contrasts TFLite and LiteRT-LM artifacts and runtime controls, then explains why transformer inference needs separate prefill and decode measurements.
- Transformer bundle and streaming interface: Describe how a LiteRT-LM bundle and its app interface differ from a TFLite file and Interpreter. Explain why the bundle includes tokenizer, KV-cache, and sampling metadata.
Why it matters: The artifact and API must match the kind of workload the app runs. - Delegate binding versus backend hint: Distinguish TFLite delegate acceptance from LiteRT-LM backend selection, and explain why measured compute units must be recorded.
Why it matters: A requested CPU or GPU backend is not proof of where a LiteRT-LM workload ran. - Prefill decode and response latency: Explain what prefill and decode do, how their costs respond to prompt length, and why TTFT is the user-facing response-latency measure here.
Why it matters: Conflating first-token delay with ongoing decode speed can produce the wrong user-facing latency target.
Key terms: .tflite, .litertlm, Interpreter, tokenizer config, KV-cache layout, sampling configuration default, delegate, Backend hint, EngineConfig, compute_units
11.3 Apple silicon with Core ML and MLX
This section distinguishes Core ML and MLX deployment roles and explains how to interpret the Apple-device measurements used in the chapter.
- Core ML and MLX roles: Distinguish Core ML’s broad Apple-device coverage from MLX’s M-series-only transformer decode role, including Core ML’s use for vision, audio, and LLM prefill.
Why it matters: Choosing between them starts with the target device and workload. - Core ML compute-unit hint: Explain that Core ML’s MLComputeUnits setting is a hint that limits eligible compute units, not a guarantee that every operation runs on the Apple Neural Engine.
Why it matters: A compute-unit setting alone does not prove where every operation executes. - Apple measurement provenance and limits: Describe where the Mac and iPhone records come from and why their reported power values cannot be treated alike.
Why it matters: The source and limits of each measurement determine which device comparisons the records can support.
Key terms: Core ML, .mlpackage, MLX, prefill, autoregressive-decode, MLComputeUnits, compute-unit hint, per-op fallback, Apple Neural Engine, op-support table
11.3.1 The compute-unit dispatch knob
This section explains how Core ML compute-unit settings govern eligible hardware and why dispatch matters more than weight variant for the measured EfficientNet-Lite0 workload.
- Core ML compute-unit choices: Explain what each MLComputeUnits setting permits and why a compute-unit hint does not necessarily dictate where an eligible operation runs.
Why it matters: The setting determines which accelerators Core ML can consider before a weight format can affect performance. - Dispatch versus weight variant: Explain what the EfficientNet-Lite0 latency comparison shows about prioritizing compute-unit dispatch over weight-variant choice, including the different CPU, GPU, and ANE patterns.
Why it matters: A faster weight format may matter less than allowing the model to reach the appropriate compute unit. - Paired iPhone measurement interpretation: Explain why the paired-iPhone Xcode result uses all, and why its similarity to the M3 ANE-permitted result does not imply equal Neural Engine capability.
Why it matters: Similar latency on a small model should not be mistaken for evidence that two chips have identical peak capability.
Key terms: MLComputeUnits, cpuOnly, cpuAndGPU, cpuAndNeuralEngine, all, compute-unit hint, weight variant, INT8 weight quantization, ANE, Xcode Performance Reports
11.3.2 The ANE op-support cliff
This section explains why unsupported transformer operators keep iPhone Llama models off the Apple Neural Engine, how to verify their actual routing, and what the Core ML Llama API provides.
- Operator support and silent fallback: Explain how Neural Engine operator support determines execution routing and how to check where a model actually runs.
Why it matters: A requested compute unit does not prove that the model ran there. - Llama quantization and ANE coverage: Explain why changing the tested iPhone Llama weight formats does not move its execution onto the Neural Engine.
Why it matters: Smaller weights alone cannot overcome an operator-support mismatch. - Core ML Llama API limits: Distinguish the measurable Core ML iPhone Llama path from an externally accessible autoregressive decoding or production deployment path.
Why it matters: A measured Core ML prefill result should not be presented as an externally reproducible autoregressive decode result.
Key terms: Neural Engine, supported-op table, compute-unit hint, GPU fallback, Xcode Performance Report, MLComputePlan, Llama, ANE coverage, INT4 per-block, INT8 per-block
11.3.3 MLX as Core ML's transformer sibling
This section contrasts Apple silicon runtime paths for vision and transformer workloads, then explains how weight quantization affects MLX decode throughput.
- Apple runtime accelerator access: Distinguish which accelerators Core ML, PyTorch MPS, and MLX can reach, and explain why routing a vision model through Metal rather than the Neural Engine incurs a large latency cost.
Why it matters: Choosing a runtime determines which hardware can execute the model, not just which API the application uses. - MLX transformer deployment role: Explain why this section positions MLX for autoregressive decode rather than Neural Engine vision inference, and distinguish a PyTorch MPS research run from the Core ML production path.
Why it matters: The appropriate Apple runtime depends on the workload and its actual deployment path. - Quantized decode bandwidth limits: Explain why four-bit MLX weights improve short-prompt decode throughput and why that advantage narrows with a longer prompt.
Why it matters: A smaller model can decode faster, but its speedup depends on what else consumes memory bandwidth.
Key terms: Core ML, PyTorch MPS, MLX, Metal, Neural Engine, compute-unit hint, autoregressive decode, coremltools.convert(...), .mlpackage, memory-bandwidth-bound
11.3.4 Sustained throughput and the palettize cliff
This section shows why sustained throughput, weight format, and actual operator routing must be measured rather than inferred from short runs, file size, or device settings.
- Workload dependent sustained throughput: Explain why a thermally representative late-window throughput measurement is more useful than an initial burst for a continuously running workload, and why throughput can fall, rise, or stay flat over the same window.
Why it matters: A fast first burst can give the wrong impression of the rate a continuous workload will deliver. - Palettized weight latency cliff: Explain why palettized weights made the small Whisper-tiny encoder slower on the Neural Engine despite a smaller weight file, and describe how to check whether that tradeoff applies to another model.
Why it matters: A smaller model file can run substantially slower if its weight format adds costly work to each operator. - Actual Apple operator routing: Distinguish the weight variant, compute-unit hint, and graph runtime on Apple silicon, and explain why actual operator routing must be verified.
Why it matters: A compute-unit setting does not prove where a model's operations actually ran.
Key terms: steady-state, soft-throttle, last-30-second throughput, memory-bandwidth-bound, palettize_weights, palette, lookup-table decompression, variant × device sweep, weight variant, MLComputeUnits
11.4 Phone, SBC, and small-box tiers
This section explains how deployment surface determines the phone, SBC, or small-box tier, and how each tier changes runtime choices, accelerator support, and validation work.
- Deployment Surface Tier Selection: Explain why the deployment surface, rather than the model, determines the tier. Describe when a consumer phone, a fixed installation, or a throughput-bound 24/7 workload calls for the phone, SBC, or small-box tier.
Why it matters: Choosing the tier first narrows the runtime and optimization choices to ones the deployment can actually use. - SBC Runtime and CPU Path: Describe what changes when a phone deployment moves to a typical Linux SBC, including which phone delegates disappear and which runtime and CPU delegate become the workhorse.
Why it matters: The available delegates, rather than the model file alone, determine the SBC's practical latency choices. - Attached Accelerator Op Support: Explain how an attached Coral or Hailo accelerator changes an SBC deployment and why INT8 conversion alone does not guarantee accelerator execution.
Why it matters: Checking operation support prevents an apparently compatible INT8 model from falling back to the CPU and missing throughput targets. - Small-Box Vendor Runtime Lock-In: Describe how a small-box appliance differs from the TFLite-based tiers and why targeting multiple vendors requires separate conversion and validation pipelines.
Why it matters: A compiled small-box model cannot simply be moved between vendors, so deployment plans must account for SDK-specific work. - Cross-Runtime Calibration Drift: Explain why the same source model and representative dataset can yield different INT8 accuracy across small-box vendors, and state the resulting calibration obligation.
Why it matters: Per-vendor accuracy checks catch quality changes that a single successful model calibration would miss.
Key terms: deployment surface, phone tier, SBC tier, small-box tier, LiteRT on Linux, libtensorflowlite.so, XNNPACK, .tflite flatbuffer, NNAPI, Edge TPU compiler
11.5 Pre and post-processing
Shows why edge-device latency must include pre- and post-processing, and how platform-native implementations can change which stage is worth optimizing.
- End-to-end latency accounting: Explain why inference-only timing is useful for comparing runtimes but insufficient for a user-facing SLA. State that the SLA should report end-to-end p95, with inference-only latency alongside it as a diagnostic.
Why it matters: A fast model call can still produce a slow user experience. - Platform-dependent pipeline bottlenecks: Describe how the measured audio and vision preprocessing costs change optimization priorities across devices. Use the Pixel log-mel and M3 Python-versus-vImage vision examples to explain why inference-only rankings can mislead.
Why it matters: The largest latency reduction may come from changing preprocessing rather than changing the model variant. - Native preprocessing remedies: Explain the section’s preferred ways to reduce pre- and post-processing latency: use platform-native primitives when available, and for a missing primitive such as Android log-mel, put the operations in the model graph or bind a tuned native library.
Why it matters: Choosing the right implementation of a surrounding pipeline stage can save more time than tuning inference. - LLM preprocessing focus: Distinguish inexpensive tokenization on the measured short prompt from the LLM preparation work that can dominate time to first token in production.
Why it matters: For LLMs, the stage worth measuring may be application-level prompt and tool handling rather than tokenization.
Key terms: inference-only, end-to-end p95, pre/post, SLA, log-mel, Kotlin radix-2, PIL, OpenCV, vImage, platform-native primitives
11.6 Summary
Where this is used later
- Ship a sub-second vision pipeline on device hardware (chapter 12) · The distinction between short-run speed and sustained throughput at a thermal knee.
- Compare the device and the server without faking the axis (chapter 12) · The distinction between estimated and sensor-grade power numbers.
- Compare throughput and latency across formats (chapter 12) · Supports interpreting throughput and latency comparisons for token-generating workloads.
Concepts explained in this chapter
- time to first tokenThe elapsed time from submitting a language-model request until its first generated token is available.
- SBC tierThe SBC tier is the deployment class based on single-board computers, with optional attached accelerators.
- deployment surfaceA deployment surface is the target device and operating environment on which a model is delivered and executed.
- phone tierThe phone tier is the deployment class comprising smartphones with integrated processing hardware and mobile-device constraints.
- small-box tierThe small-box tier is the deployment class for sustained workloads on fixed installations with discrete inference accelerators.
- delegateA delegate is a runtime component that executes supported model operations through a particular hardware backend or optimized operator library.
- mean powerMean power is the average electrical power consumed by a device or workload over a measurement interval.
- LiteRT-LMLiteRT-LM is a runtime for deploying and executing transformer models, with bundle-based inputs and streaming generation APIs.
- energy per inferenceThe electrical energy consumed to complete one model inference, commonly measured in joules.
- dynamic quantizationDynamic range quantization stores model weights at reduced precision and handles activation ranges dynamically during inference.
- latencyThe elapsed time required for a model or system to produce a result.
- log-mel featuresA log-mel STFT is a short-time Fourier transform followed by conversion of spectral magnitudes to a logarithmic mel-frequency representation.
- percentileA quantile is a value that divides a probability distribution so that a specified proportion of observations falls below it.
- thermal kneeA thermal knee is the point during sustained computation at which thermal management causes a sharp reduction in performance.
- cross-runtime calibration driftCross-runtime calibration drift is a difference in quantization calibration results produced by distinct runtimes or compilers for the same model and input data.
- tier selectionTier selection is the choice of deployment class based on the constraints and capabilities of the target deployment surface.
- compute-unit hintA compute-unit hint is a Core ML setting that limits which compute units the runtime may consider without dictating operation placement.
- Apple MLXMLX is an array framework for machine-learning research and development, designed for efficient execution on Apple silicon.
- palettizationWeight clustering groups neural-network weights around shared centroid values and represents the weights by cluster indices.
- .litertlm bundleA LiteRT-LM bundle is a model artifact in the FlatBuffers family that packages a transformer model with runtime metadata.
- throughput retentionSustained throughput retention is the fraction of initial throughput maintained during extended operation under thermal load.
- int16x8A quantization configuration that uses 16-bit integer activations and 8-bit integer weights.
- preprocessing and postprocessingPre- and post-processing are the operations that transform model inputs before inference and model outputs afterward.
- powerThe rate at which a device or system uses electrical energy.
- get_tensor_detailsA TensorFlow Lite interpreter method that returns metadata about the interpreter’s tensors.
- thermal throttlingSkin-temperature throttling reduces processor operating frequency or capacity when device-surface temperature reaches a thermal limit.
- throughputThe amount of computation or output completed per unit of time, including its retention during sustained operation.
- latency budgetA latency budget is a maximum allowable elapsed time for a computation or service operation.
- TFLiteConverterTFLiteConverter is the TensorFlow Lite API for converting models to TensorFlow Lite format.
- powermetricspowermetrics is a macOS command-line utility that reports system and processor performance and power-related measurements.
- Core MLCore ML is Apple's framework for integrating machine-learning models into applications across its platforms.
- conversion pipelineA conversion pipeline is a sequence of processing steps that transforms a model into a format or implementation suitable for a target runtime or accelerator.
- tokenizer configurationA tokenizer configuration specifies vocabulary, special-token IDs, and normalization rules for converting text into model tokens.
- weight-only quantizationReducing the precision of model weights while leaving activations at higher precision.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
