liveBook · Manning

AMX-INT8

As explained in Quantization and Fast Inference

Intel Advanced Matrix Extensions (AMX) are processor extensions that accelerate matrix operations using tile registers and dedicated instructions.

AMX-INT8 adds a dedicated matrix-multiply unit to Intel CPUs. It has its own tile registers and instructions, including 16×16 INT8 tile multiplication. section 10.4 section 10.4.1

In llama.cpp, AMX kernels must be compiled in with build flags such as GGML_AMX_TILE; CPU support alone does not enable them. Runtime dispatch uses AMX for sufficiently large matrix shapes and falls back to AVX-512 below an internal threshold. section 10.4.1

Table 10.1 Compiled ISA features per build
Feature
build-avx2
build-avx512
build-avx512-amx
Source location
AVX2
ON
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AVX-512 F
OFF
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AVX-512 VNNI
OFF
ON
ON
ggml/src/ggml-cpu/arch/x86/quants.c
AMX-INT8
OFF
OFF
ON
ggml/src/ggml-cpu/amx/mmq.cpp

The build-feature table shows AMX-INT8 enabled only in the AMX build, alongside AVX2 and AVX-512 features. section 10.4.1

AMX performance depends on request shape:

  • For single-stream decode, weight-memory bandwidth limits throughput. On the tested Llama-2-7B setup, the AMX build was slower than AVX-512 across quantized formats, with regressions from 6.3% to 28.5%. section 10.4.3
  • For prompt processing above roughly 256 input tokens, AMX can win because larger matrices amortize tile loads across the activation dimension. section 10.4.5
  • Compile separate binaries and route requests by shape when both workloads matter; otherwise, choose the tradeoff that fits the deployment. section 10.4.5

How Quantization and Fast Inference covers it

How deep it goes
Explains tile registers, tile operations, build inclusion, and shape-based dispatch, then measures workload-specific throughput.
Examples
Compares llama.cpp builds on Sapphire Rapids and reports decode regressions alongside prompt-processing gains.
What it assumes
Assumes awareness of CPU feature support and teaches that build flags determine whether the binary contains AMX kernels.
What you can do after
Operate/configure it by compiling AMX kernels and routing requests by shape when decode and prompt workloads differ.
How it connects
Connects AMX-INT8 to AVX-512, VNNI, and llama.cpp's quantized CPU kernel dispatch.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.