AMX-INT8 adds a dedicated matrix-multiply unit to Intel CPUs. It has its own tile registers and instructions, including 16×16 INT8 tile multiplication. section 10.4 section 10.4.1
In llama.cpp, AMX kernels must be compiled in with build flags such as GGML_AMX_TILE; CPU support alone does not enable them. Runtime dispatch uses AMX for sufficiently large matrix shapes and falls back to AVX-512 below an internal threshold. section 10.4.1
Table 10.1 Compiled ISA features per build
|
Feature
|
build-avx2
|
build-avx512
|
build-avx512-amx
|
Source location
|
|
AVX2
|
ON
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AVX-512
F
|
OFF
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AVX-512
VNNI
|
OFF
|
ON
|
ON
|
ggml/src/ggml-cpu/arch/x86/quants.c
|
|
AMX-INT8
|
OFF
|
OFF
|
ON
|
ggml/src/ggml-cpu/amx/mmq.cpp
|
The build-feature table shows AMX-INT8 enabled only in the AMX build, alongside AVX2 and AVX-512 features. section 10.4.1
AMX performance depends on request shape:
- For single-stream decode, weight-memory bandwidth limits throughput. On the tested Llama-2-7B setup, the AMX build was slower than AVX-512 across quantized formats, with regressions from 6.3% to 28.5%. section 10.4.3
- For prompt processing above roughly 256 input tokens, AMX can win because larger matrices amortize tile loads across the activation dimension. section 10.4.5
- Compile separate binaries and route requests by shape when both workloads matter; otherwise, choose the tradeoff that fits the deployment. section 10.4.5
