AVX-512 doubles x86 vector-register width to 512 bits and adds masked operations. For CPU inference, its practical value depends on the workload: wider vectors help only when computation, rather than memory bandwidth, limits throughput. section 10.4 section 10.4.3
In llama.cpp, build-time flags determine whether AVX-512 kernels are compiled into the binary; CPU support alone does not enable them. The measured AVX-512 build also enables VNNI, whose fused integer dot-product instruction targets quantized matrix multiplication. section 10.4.1 section 10.4
Decode throughput
On one Sapphire Rapids host, AVX-512 with VNNI improves integer-format single-token decode by 1.8–3.7% over AVX-2. The workload is bandwidth-bound; prompt processing can show larger gains when matrix dimensions amortize loads. section 10.4.3
Choose the build for the request shape:
- For single-stream Llama-2-7B decode on Sapphire Rapids, use AVX-512 with VNNI and leave AMX off; the measured AMX build regresses on quantized formats. section 10.4.5
- For prompt processing above roughly 256 tokens, AMX can win; consider separate binaries routed by request shape. section 10.4.5
Table 10.1 Compiled ISA features per build
|
Feature
|
build-avx2
|
build-avx512
|
build-avx512-amx
|
Source location
|
|
AVX2
|
ON
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AVX-512
F
|
OFF
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AVX-512
VNNI
|
OFF
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AMX-INT8
|
OFF
|
OFF
|
ON
|
ggml/src/ggml-cpu/amx/mmq.cpp
|
Compare which ISA features each build compiles in. section 10.4.1
