liveBook · Manning

GGUF

As explained in Quantization and Fast Inference

GGUF is a file format for packaging quantized machine-learning models and their associated metadata for inference runtimes.

GGUF is a self-contained binary container for CPU inference with llama.cpp. It stores model architecture, tokenizer, weights, and metadata in one file, with quantized tensors arranged in fixed layouts that inference kernels consume directly. section 10.2 section 10.0

GGUF quantization is a separate ecosystem from ONNX operator-level quantization: its layouts do not round-trip through ONNX. K-quants such as Q4_K_M use superblocks with shared scales; i-quants such as IQ4_XS use importance-calibrated codebooks. section 10.6 section 10.0

A typical Hugging Face-to-GGUF conversion proceeds through these stages:

  • Convert the checkpoint to FP16 GGUF with convert_hf_to_gguf.py, then use llama-quantize to write a target format such as Q4_K_M or IQ4_XS. section 10.6
  • For the IQ path, generate an importance matrix with llama-imatrix before fitting the codebook. section 10.6

A 7B model’s file footprint

A llama-2-7b Q4_K_M GGUF is 3.80 GB, compared with 6.67 GB for Q8_0. The quantized file size matters for CPU decode because generated tokens require reading model weights from memory. section 10.2 section 10.6

How Quantization and Fast Inference covers it

How deep it goes
Explains the self-contained container, fixed tensor layouts, K-quant superblocks, and importance-calibrated i-quant codebooks.
Examples
Uses llama-2-7b Q4_K_M and Q8_0 files, reporting sizes of 3.80 GB and 6.67 GB.
What you can do after
Build CPU-ready GGUF artifacts by converting Hugging Face checkpoints and quantizing them with llama.cpp tools.
How it connects
Connects GGUF to llama.cpp, GGML, K-quant, i-quant, ONNX, and Hugging Face checkpoint conversion.

Also explained in

Compare how 4 books explain GGUF →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.