CPU-first LLM inference can suit deployments driven by cost, fleet bin-packing, or the absence of GPUs. Throughput depends on matching quantization, thread count, and compiled kernels to the workload and hardware; misconfiguration can leave 10–30% of potential throughput unused. section 10.0
In llama.cpp, CMake ISA flags determine which CPU kernel paths exist in the binary. Hardware support alone is not enough: a build without AVX-512 code cannot use those instructions, even on a CPU that supports them. AMX kernels add a separate path and fall back to AVX-512 when matrix dimensions are too small to amortize tile-load overhead. section 10.4.1
Table 10.1 Compiled ISA features per build
|
Feature
|
build-avx2
|
build-avx512
|
build-avx512-amx
|
Source location
|
|
AVX2
|
ON
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AVX-512
F
|
OFF
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AVX-512
VNNI
|
OFF
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AMX-INT8
|
OFF
|
OFF
|
ON
|
ggml/src/ggml-cpu/amx/mmq.cpp
|
Compare which ISA features are compiled into three llama.cpp builds. section 10.4.1








