liveBook · Manning

AVX-512

As explained in Quantization and Fast Inference

AVX-512 is a family of x86 instruction-set extensions providing 512-bit SIMD operations and related instruction subsets.

AVX-512 doubles x86 vector-register width to 512 bits and adds masked operations. For CPU inference, its practical value depends on the workload: wider vectors help only when computation, rather than memory bandwidth, limits throughput. section 10.4 section 10.4.3

In llama.cpp, build-time flags determine whether AVX-512 kernels are compiled into the binary; CPU support alone does not enable them. The measured AVX-512 build also enables VNNI, whose fused integer dot-product instruction targets quantized matrix multiplication. section 10.4.1 section 10.4

Decode throughput

On one Sapphire Rapids host, AVX-512 with VNNI improves integer-format single-token decode by 1.8–3.7% over AVX-2. The workload is bandwidth-bound; prompt processing can show larger gains when matrix dimensions amortize loads. section 10.4.3

Choose the build for the request shape:

  • For single-stream Llama-2-7B decode on Sapphire Rapids, use AVX-512 with VNNI and leave AMX off; the measured AMX build regresses on quantized formats. section 10.4.5
  • For prompt processing above roughly 256 tokens, AMX can win; consider separate binaries routed by request shape. section 10.4.5
Table 10.1 Compiled ISA features per build
Feature
build-avx2
build-avx512
build-avx512-amx
Source location
AVX2
ON
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AVX-512 F
OFF
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AVX-512 VNNI
OFF
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AMX-INT8
OFF
OFF
ON
ggml/src/ggml-cpu/amx/mmq.cpp

Compare which ISA features each build compiles in. section 10.4.1

How Quantization and Fast Inference covers it

How deep it goes
Explains vector capabilities, build-time kernel selection, and measured workload tradeoffs without detailing instruction implementations.
Examples
Compares Llama-2-7B decode throughput across AVX-2, AVX-512 with VNNI, and AMX-enabled builds.
What you can do after
Configure llama.cpp builds and select ISA options according to CPU support and request shape.
How it connects
Connects AVX-512 to AVX2, VNNI, AMX, SIMD, and quantized matrix multiplication.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.