MLX targets autoregressive transformer decoding on M-series Macs. It runs on the CPU and GPU through its own Metal kernels, but does not access the Neural Engine or provide vision kernels. Choose it for transformer workloads, not as a general replacement for Core ML. section 11.3.3 section 11.6
For decoder inference, MLX supports these weight formats and tradeoffs:
- MLX converts Hugging Face checkpoints to fp16, q4, or q8 formats. Quantization reduces weight bandwidth, a key constraint during decode. section 11.3.3
- Group size 64 is the default; group size 128 uses fewer scale tensors and can suit memory pressure. section 11.3.3
Llama-3.2-1B decode on M3
At a 32-token prompt, q4 with group size 128 reaches 78.5 tokens/s, versus 29.5 for fp16. At 1024 tokens, q4 falls to 50.4 tokens/s; fp16 reaches 25.2. section 11.3.3
Figure 11.11 Llama-3.2-1B autoregressive decode throughput, three M3 MLX variants × three prompt lengths plus Pixel 10 Pro LiteRT-LM INT8 baseline. M3 mlx_q4_g128 peaks at 78.5 tok/s at 32-token prompt and falls to 50.4 tok/s at 1024 tokens. M3 mlx_q8_g128 reads 54.6 / 48.7 / 36.2 tok/s. M3 mlx_fp16 reads 29.5 / 29.7 / 25.2 tok/s. Pixel 10 Pro LiteRT-LM INT8 reads 23.8 / 24.9 / 25.7 tok/s, flat across prompt lengths because the Tensor G5's PowerVR DXT-48-1536 GPU and its memory subsystem hit a bandwidth ceiling at every length. iPhone 16 Core ML Llama cell empty, prefill-only conversion path.

Compare M3 MLX formats across prompt lengths and the Pixel 10 Pro baseline. section 11.3.3
