liveBook · Manning

Palettization

As explained in Quantization and Fast Inference

Weight clustering groups neural-network weights around shared centroid values and represents the weights by cluster indices.

Palettization reduces weight storage by grouping weights around shared centroid values and storing an index for each weight. In Core ML, k-means produces a palette for each weight tensor; a 4-bit index addresses 16 centroids, while a 6-bit index addresses 64. The centroids retain full precision, and lookup materializes weights as FP16 before multiplication. Thus, the bit count describes the index, not the value precision used by the matmul. section 11.1.1

Treat palettization as a model- and hardware-dependent tradeoff:

  • Smaller weight files do not guarantee lower latency or power. On M3 vision variants, 4-bit palettization reads a quarter of the FP16 weight bytes, yet draws 7% more power; latency stays within 0.13 ms across the tested variants. section 11.1.1
  • Lookup overhead can dominate small operators. On iPhone 16, Whisper-tiny palettized variants take 33–35 ms versus about 18 ms for linear-quantized variants; larger Stable Diffusion UNet or Llama MLP blocks are better candidates. section 11.3.4

Choose by workload, not file size

Sweep weight variants on the target device and workload. A format that works for a large operator may lose on a small audio encoder, where table lookups consume a larger share of execution time. section 11.3.4

Figure 11.1 Cross-modal design space on MacBook Air M3 (laptop) and Google Pixel 10 Pro (phone). Marker shape encodes modality (circle = vision, triangle = audio, square = LLM). Twelve points populate the chart. Five M3 vision variants sit at 4.4–6.6 W, two M3 audio variants at 5.7–6.7 W, Pixel vision at 1.75 W, Pixel audio at 3.33 W, three Pixel LLM points at 5.51 W. The iPhone 16 is absent. iOS exposes no third-party API for instantaneous power (MetricKit reports daily-aggregated CPU and GPU time, not a milliwatt figure at the cadence this axis needs). M3 LLM is also absent. MLX publishes neither TTFT nor power telemetry, so the cell cannot land on either axis. Latency is p50 for vision and audio; TTFT for LLM. Power is powermetrics on macOS and BatteryManager on Android, both over a 30 s window.

Compare measured latency and steady-state power across vision, audio, and LLM workloads on M3 and Pixel devices. section 11.1.1

How Quantization and Fast Inference covers it

How deep it goes
Explains k-means palettes, index widths, lookup-based reconstruction, and operator-level performance consequences.
Examples
Uses Core ML vision variants, Whisper-tiny audio encoders, and Stable Diffusion or Llama blocks.
What you can do after
Evaluate palettized variants with device-and-workload sweeps, weighing weight-file size against latency and power.
How it connects
Connects weight clustering to linear quantization, Core ML, and Neural Engine execution.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.