liveBook · Manning

Apple MLX

As explained in Quantization and Fast Inference

MLX is an array framework for machine-learning research and development, designed for efficient execution on Apple silicon.

MLX targets autoregressive transformer decoding on M-series Macs. It runs on the CPU and GPU through its own Metal kernels, but does not access the Neural Engine or provide vision kernels. Choose it for transformer workloads, not as a general replacement for Core ML. section 11.3.3 section 11.6

For decoder inference, MLX supports these weight formats and tradeoffs:

  • MLX converts Hugging Face checkpoints to fp16, q4, or q8 formats. Quantization reduces weight bandwidth, a key constraint during decode. section 11.3.3
  • Group size 64 is the default; group size 128 uses fewer scale tensors and can suit memory pressure. section 11.3.3

Llama-3.2-1B decode on M3

At a 32-token prompt, q4 with group size 128 reaches 78.5 tokens/s, versus 29.5 for fp16. At 1024 tokens, q4 falls to 50.4 tokens/s; fp16 reaches 25.2. section 11.3.3

Figure 11.11 Llama-3.2-1B autoregressive decode throughput, three M3 MLX variants × three prompt lengths plus Pixel 10 Pro LiteRT-LM INT8 baseline. M3 mlx_q4_g128 peaks at 78.5 tok/s at 32-token prompt and falls to 50.4 tok/s at 1024 tokens. M3 mlx_q8_g128 reads 54.6 / 48.7 / 36.2 tok/s. M3 mlx_fp16 reads 29.5 / 29.7 / 25.2 tok/s. Pixel 10 Pro LiteRT-LM INT8 reads 23.8 / 24.9 / 25.7 tok/s, flat across prompt lengths because the Tensor G5's PowerVR DXT-48-1536 GPU and its memory subsystem hit a bandwidth ceiling at every length. iPhone 16 Core ML Llama cell empty, prefill-only conversion path.

Compare M3 MLX formats across prompt lengths and the Pixel 10 Pro baseline. section 11.3.3

How Quantization and Fast Inference covers it

How deep it goes
Describes Metal execution, checkpoint formats, group sizes, and measured decode throughput, but not vision kernels or Neural Engine execution.
Examples
Compares q4, q8, and fp16 Llama-3.2-1B decode throughput across prompt lengths on M3.
What it assumes
Assumes an M-series Mac for MLX execution; no further background requirements are specified.
What you can do after
Evaluate decode-throughput tradeoffs among MLX quantization formats and group sizes on M-series hardware.
How it connects
Connects transformer decoding and weight quantization with Metal, Hugging Face checkpoints, and Core ML.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.