Palettization reduces weight storage by grouping weights around shared centroid values and storing an index for each weight. In Core ML, k-means produces a palette for each weight tensor; a 4-bit index addresses 16 centroids, while a 6-bit index addresses 64. The centroids retain full precision, and lookup materializes weights as FP16 before multiplication. Thus, the bit count describes the index, not the value precision used by the matmul. section 11.1.1
Treat palettization as a model- and hardware-dependent tradeoff:
- Smaller weight files do not guarantee lower latency or power. On M3 vision variants, 4-bit palettization reads a quarter of the FP16 weight bytes, yet draws 7% more power; latency stays within 0.13 ms across the tested variants. section 11.1.1
- Lookup overhead can dominate small operators. On iPhone 16, Whisper-tiny palettized variants take 33–35 ms versus about 18 ms for linear-quantized variants; larger Stable Diffusion UNet or Llama MLP blocks are better candidates. section 11.3.4
Choose by workload, not file size
Sweep weight variants on the target device and workload. A format that works for a large operator may lose on a small audio encoder, where table lookups consume a larger share of execution time. section 11.3.4
Figure 11.1 Cross-modal design space on MacBook Air M3 (laptop) and Google Pixel 10 Pro (phone). Marker shape encodes modality (circle = vision, triangle = audio, square = LLM). Twelve points populate the chart. Five M3 vision variants sit at 4.4–6.6 W, two M3 audio variants at 5.7–6.7 W, Pixel vision at 1.75 W, Pixel audio at 3.33 W, three Pixel LLM points at 5.51 W. The iPhone 16 is absent. iOS exposes no third-party API for instantaneous power (MetricKit reports daily-aggregated CPU and GPU time, not a milliwatt figure at the cadence this axis needs). M3 LLM is also absent. MLX publishes neither TTFT nor power telemetry, so the cell cannot land on either axis. Latency is p50 for vision and audio; TTFT for LLM. Power is powermetrics on macOS and BatteryManager on Android, both over a 30 s window.

Compare measured latency and steady-state power across vision, audio, and LLM workloads on M3 and Pixel devices. section 11.1.1
