liveBook · Manning

AVX2

As explained in Quantization and Fast Inference

AVX2 is an x86 SIMD instruction-set extension that adds 256-bit integer vector operations to AVX.

AVX2 adds 256-bit vector registers to x86, allowing 32 INT8 lanes per instruction. For CPU inference, its practical value depends on whether the workload is limited by compute or memory bandwidth. section 10.4 section 10.4.3

In llama.cpp, build-time ISA flags determine which CPU kernels the binary contains. An AVX2-only build enables AVX and AVX2 while excluding richer instruction paths, even when the host supports them. section 10.4.1

Decode throughput

On one Sapphire Rapids host, AVX-512 with VNNI improved integer-format decode throughput over AVX2 by 1.8–3.7%. F16 throughput was effectively unchanged because decode was bandwidth-bound. section 10.4.3

Table 10.1 Compiled ISA features per build
Feature
build-avx2
build-avx512
build-avx512-amx
Source location
AVX2
ON
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AVX-512 F
OFF
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AVX-512 VNNI
OFF
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AMX-INT8
OFF
OFF
ON
ggml/src/ggml-cpu/amx/mmq.cpp

Compare which ISA features each llama.cpp build compiles in. section 10.4.1

How Quantization and Fast Inference covers it

How deep it goes
Defines AVX2's vector width and build role, then compares decode performance without detailing its kernel instructions.
Examples
Compares AVX2-only, AVX-512 with VNNI, and AMX-enabled builds on decode throughput.
What you can do after
Evaluate how build-time ISA selection affects decode throughput under a bandwidth-bound workload.
How it connects
Connects AVX2 builds with AVX-512, VNNI, AMX, and llama.cpp kernel selection.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.