GGUF is a self-contained binary container for CPU inference with llama.cpp. It stores model architecture, tokenizer, weights, and metadata in one file, with quantized tensors arranged in fixed layouts that inference kernels consume directly. section 10.2 section 10.0
GGUF quantization is a separate ecosystem from ONNX operator-level quantization: its layouts do not round-trip through ONNX. K-quants such as Q4_K_M use superblocks with shared scales; i-quants such as IQ4_XS use importance-calibrated codebooks. section 10.6 section 10.0
A typical Hugging Face-to-GGUF conversion proceeds through these stages:
- Convert the checkpoint to FP16 GGUF with convert_hf_to_gguf.py, then use llama-quantize to write a target format such as Q4_K_M or IQ4_XS. section 10.6
- For the IQ path, generate an importance matrix with llama-imatrix before fitting the codebook. section 10.6
A 7B model’s file footprint
A llama-2-7b Q4_K_M GGUF is 3.80 GB, compared with 6.67 GB for Q8_0. The quantized file size matters for CPU decode because generated tokens require reading model weights from memory. section 10.2 section 10.6



