End-to-end latency includes input preparation and output handling, not just the model’s forward pass. Decode, resize, and normalize images; convert audio into log-mel features; tokenize prompts. Measure these stages separately: they can rival or exceed inference time. section 11.5
Optimization priorities depend on the device and modality:
- On M3, audio log-mel processing takes about 22 ms with Accelerate.vDSP, versus 14 ms for encoder inference. On Pixel, a Kotlin log-mel implementation takes about 243 ms, close to the 290 ms inference time. section 11.5
- M3 vision preprocessing takes about 50 ms, exceeding a 30 fps camera’s 33 ms frame budget before inference. Pixel vision preprocessing takes about 5.3 ms. section 11.5
Figure 11.12 Pre/post breakdown by modality and device, log y-axis. M3 vision (Python PIL or OpenCV) reads ~30 ms decode + ~17 ms resize + ~3 ms normalize, totalling ~50 ms, already above the 33 ms frame budget for a 30 fps camera before inference begins. M3 audio reads ~11 ms decode + ~22 ms log-mel via Apple Accelerate.vDSP, totalling ~33 ms. M3 text reads ~0.8 ms tokenize + ~0.1 ms detokenize via Hugging Face Rust-backed tokenizers for a 128-token Llama prompt. Pixel vision reads ~0.4 ms decode + ~0.1 ms resize + ~4.7 ms normalize via BitmapFactory + Canvas + Kotlin normalize loop, totalling ~5.3 ms, faster than every M3 vision pipeline measured. Pixel audio reads ~243 ms log-mel via a Kotlin radix-2 STFT implementation, two orders of magnitude slower than M3 Accelerate.vDSP on the same Whisper-tiny feature pipeline.

Compare per-stage latency across vision, audio, and text pipelines on M3 and Pixel; the log-scale breakdown highlights platform-specific bottlenecks. section 11.5
