AVX2 adds 256-bit vector registers to x86, allowing 32 INT8 lanes per instruction. For CPU inference, its practical value depends on whether the workload is limited by compute or memory bandwidth. section 10.4 section 10.4.3
In llama.cpp, build-time ISA flags determine which CPU kernels the binary contains. An AVX2-only build enables AVX and AVX2 while excluding richer instruction paths, even when the host supports them. section 10.4.1
Decode throughput
On one Sapphire Rapids host, AVX-512 with VNNI improved integer-format decode throughput over AVX2 by 1.8–3.7%. F16 throughput was effectively unchanged because decode was bandwidth-bound. section 10.4.3
Table 10.1 Compiled ISA features per build
|
Feature
|
build-avx2
|
build-avx512
|
build-avx512-amx
|
Source location
|
|
AVX2
|
ON
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AVX-512
F
|
OFF
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AVX-512
VNNI
|
OFF
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AMX-INT8
|
OFF
|
OFF
|
ON
|
ggml/src/ggml-cpu/amx/mmq.cpp
|
Compare which ISA features each llama.cpp build compiles in. section 10.4.1
