liveBook · Manning

Preprocessing and postprocessing

As explained in Quantization and Fast Inference

Pre- and post-processing are the operations that transform model inputs before inference and model outputs afterward.

End-to-end latency includes input preparation and output handling, not just the model’s forward pass. Decode, resize, and normalize images; convert audio into log-mel features; tokenize prompts. Measure these stages separately: they can rival or exceed inference time. section 11.5

Optimization priorities depend on the device and modality:

  • On M3, audio log-mel processing takes about 22 ms with Accelerate.vDSP, versus 14 ms for encoder inference. On Pixel, a Kotlin log-mel implementation takes about 243 ms, close to the 290 ms inference time. section 11.5
  • M3 vision preprocessing takes about 50 ms, exceeding a 30 fps camera’s 33 ms frame budget before inference. Pixel vision preprocessing takes about 5.3 ms. section 11.5
Figure 11.12 Pre/post breakdown by modality and device, log y-axis. M3 vision (Python PIL or OpenCV) reads ~30 ms decode + ~17 ms resize + ~3 ms normalize, totalling ~50 ms, already above the 33 ms frame budget for a 30 fps camera before inference begins. M3 audio reads ~11 ms decode + ~22 ms log-mel via Apple Accelerate.vDSP, totalling ~33 ms. M3 text reads ~0.8 ms tokenize + ~0.1 ms detokenize via Hugging Face Rust-backed tokenizers for a 128-token Llama prompt. Pixel vision reads ~0.4 ms decode + ~0.1 ms resize + ~4.7 ms normalize via BitmapFactory + Canvas + Kotlin normalize loop, totalling ~5.3 ms, faster than every M3 vision pipeline measured. Pixel audio reads ~243 ms log-mel via a Kotlin radix-2 STFT implementation, two orders of magnitude slower than M3 Accelerate.vDSP on the same Whisper-tiny feature pipeline.

Compare per-stage latency across vision, audio, and text pipelines on M3 and Pixel; the log-scale breakdown highlights platform-specific bottlenecks. section 11.5

How Quantization and Fast Inference covers it

How deep it goes
Breaks vision, audio, and text pipelines into timed stages and compares device costs.
Examples
Compares M3 and Pixel timings for vision, audio, and text preprocessing and postprocessing.
What you can do after
Diagnose end-to-end latency by timing preprocessing and postprocessing separately from model inference.
How it connects
Connects inference latency with image decoding, resizing, normalization, log-mel feature extraction, and tokenization.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.