liveBook · Manning

CPU

As explained in Quantization and Fast Inference

A CPU is a general-purpose processor that executes program instructions and arithmetic operations.

CPU-first LLM inference can suit deployments driven by cost, fleet bin-packing, or the absence of GPUs. Throughput depends on matching quantization, thread count, and compiled kernels to the workload and hardware; misconfiguration can leave 10–30% of potential throughput unused. section 10.0

In llama.cpp, CMake ISA flags determine which CPU kernel paths exist in the binary. Hardware support alone is not enough: a build without AVX-512 code cannot use those instructions, even on a CPU that supports them. AMX kernels add a separate path and fall back to AVX-512 when matrix dimensions are too small to amortize tile-load overhead. section 10.4.1

Table 10.1 Compiled ISA features per build
Feature
build-avx2
build-avx512
build-avx512-amx
Source location
AVX2
ON
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AVX-512 F
OFF
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AVX-512 VNNI
OFF
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AMX-INT8
OFF
OFF
ON
ggml/src/ggml-cpu/amx/mmq.cpp

Compare which ISA features are compiled into three llama.cpp builds. section 10.4.1

How Quantization and Fast Inference covers it

How deep it goes
Explains how ISA flags include or exclude kernels and when AMX dispatches to AVX-512.
Examples
Compares AVX2, AVX-512/VNNI, and AMX-INT8 builds against Sapphire Rapids CPU features.
What you can do after
Configure llama.cpp builds with selected ISA kernels and verify which features were compiled in.
How it connects
Connects CPU inference to llama.cpp, GGUF, quantization, AVX2, AVX-512, and AMX.

Also explained in

Compare how 9 books explain CPU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.