liveBook · Manning

Chapter 10: Running on CPUs and Using GGUF

What you will learn in chapter 10 of Quantization and Fast Inference

This introduction distinguishes CPU-first GGUF inference from graph-based deployment and explains why silent throughput losses matter in production.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction distinguishes CPU-first GGUF inference from graph-based deployment and explains why silent throughput losses matter in production.

Key terms: GGUF, K-quant, IQ-quant, operator-level INT8 dispatch, single-binary C++ inference engine, throughput, AMX build flag, batch-1 decode, thread count, CPU replica

10.1 Know when CPU wins on cost and scale-out

This section compares CPU and GPU LLM inference by showing how utilization, latency, concurrency, memory capacity, and operational fit change the deployment decision.

Key terms: bandwidth-bound, decode, weight matrix, byte layout, dollars per million generated tokens, utilization, TPS ceiling, replicas, prefill, latency-to-first-token

10.2 Understand the GGUF format and its variants

Introduces GGUF as a self-contained model file whose quantized tensor blocks can be consumed directly by llama.cpp inference kernels.

Key terms: GGUF, GGML, llama.cpp, quantized tensor, fixed block layout

10.2.1 Three quant families, three generations

This subsection explains how legacy, K-quant, and IQ-quant GGUF formats compress weights differently, and the resulting quality and CPU-speed trade-offs.

Key terms: legacy quants, Q4_0, Q8_0, FP16 scale factor, k-quants, superblock, super-scale, sub-scales, bits per weight, S variant

10.2.2 Four sections of the GGUF file

Explains the GGUF file layout and how memory mapping, sequential weight access, and alignment support CPU inference.

Key terms: header, metadata key-value section, tensor descriptor table, packed tensor data, little-endian, u32, u64, GGUF version, tensor count, metadata key-value count

10.2.3 Metadata makes the file self-describing

This section explains how embedded GGUF metadata makes a model file self-describing, while limiting which runtimes can serve it.

Key terms: metadata section, typed key-value pairs, namespaced names, quantization recipe label, attention KV cache, manifest-sidecar pattern, format lock-in, re-export

10.2.4 The packing is not uniform

This section explains how a GGUF file records each tensor’s format and why a Q4_K_M file contains a mix of quantization types.

Key terms: tensor descriptor table, ggml type code, byte offset, per-tensor type codes, Q4_K_M, Q4_K, Q6_K, F32

10.2.5 Inside one block: from bytes to a weight

This section shows how raw bytes in Q8_0 and Q6_K GGUF blocks become reconstructed weights, and why a GGUF recipe need not use one format throughout the file.

Key terms: Q8_0, FP16 scale, signed INT8s, Q6_K, superblock, ql, qh, sub-scale, super-scale, Q4_K_M

10.3 Convert large language models to CPU-friendly artifacts

Introduces the GGUF conversion pipeline and identifies the single hardware and software run behind the results that follow.

10.3.1 The pipeline at a glance

This section maps the Hugging Face-to-GGUF conversion pipeline to its Python and C++ tools and identifies three first-run pitfalls.

Key terms: safetensors shards, convert_hf_to_gguf.py, F16 GGUF, llama-quantize, llama-imatrix, llama-completion, meta-llama/Llama-2-7b-hf, community license, huggingface-cli login, gate access

10.3.2 F16 export and the four quantization runs

This subsection explains how one F16 GGUF export feeds four quantization runs and why their one-time conversion costs differ.

Key terms: convert_hf_to_gguf.py, safetensors index, GGUF, F16, F32, llama-quantize, Q8_0, Q5_K_M, Q4_K_M, Q6_K

10.3.3 The imatrix is the IQ-quant prerequisite

This subsection explains how an imatrix calibrates IQ4_XS quantization and why its tensor-upgrade choices produce a smaller file than Q4_K_M.

Key terms: imatrix, llama-imatrix, llama-quantize, F16 model, sidecar GGUF, per-tensor activation magnitudes, IQ4_XS, Q4_K_M, Q6_K, upgrade footprint

10.3.4 Verify size, perplexity, fluency, and decode throughput

This section verifies five GGUF variants and uses size, perplexity, generated text, and decode speed to weigh their deployment trade-offs.

Key terms: gguf.GGUFReader, general.file_type, LlamaFileType, size, perplexity, fluency, IQ4_XS, Q4_K_M, imatrix, standard-error band

10.4 Compare kernel families and what they imply

This section explains why quantization-format rankings may change across x86 kernel families and sets up a controlled comparison of those families.

Key terms: IQ4_XS, Q4_K_M, codebook-lookup decode, x86, ISA, AVX-2, AVX-512, VNNI, AMX, kernel families

10.4.1 Three builds, one CPU

This subsection shows how three llama.cpp builds on the same CPU expose different instruction kernels, and why AMX execution also depends on matrix size.

Key terms: build-time ISA flags, GGML_AVX2, GGML_AVX512, GGML_AMX_TILE, build-avx2, build-avx512, AMX, AMX-INT8, AVX-512, tile-load overhead

10.4.2 Inside one Q4_K dot product

This section traces the stages of a Q4_K dot-product kernel and explains how its memory access and instruction choices affect CPU inference.

Key terms: scale broadcast, nibble unpack, fused dot, horizontal reduce, superblock, Q4_K_M, scatter-gathers, DRAM-to-L2 bandwidth, ALU rate, IQ4_XS

10.4.3 Decode throughput across the matrix

This section compares single-token decode throughput across quantization formats and CPU builds, explaining why memory bandwidth limits ISA gains and why AMX can slow decode.

Key terms: decode, bandwidth-bound, weight footprint, F16, AVX-2, AVX-512, VNNI, single-token decode, prompt processing, AMX

10.4.4 The IQ4_XS / Q4_K_M crossover

This section compares IQ4_XS and Q4_K_M decode throughput across CPU kernel families, explains why their speed gap narrows with AMX, and puts that effect in perspective.

Key terms: IQ4_XS, Q4_K_M, decode throughput, AVX-2, AVX-512+VNNI, AVX-512+AMX, dense INT8 multiply-accumulate, codebook gather, TDPBSSD, kernel-family effect

10.4.5 What to ship

This section recommends CPU builds and quantization formats for Llama-2-7B deployment, distinguishing single-stream decode from prompt processing.

Key terms: single-stream decode, prompt processing, AVX-512, VNNI, AMX, IQ4_XS, Q4_K_M, 16-entry codebooks, decode throughput, build-time CMake flags

10.5 Set realistic expectations for throughput and memory

This section challenges two CPU deployment assumptions: that model file size determines RAM requirements and that one thread per physical core always gives the best decode throughput.

Key terms: Q4_K_M, RAM, OOM, -t, physical core, Simultaneous Multi-Threading (SMT), bandwidth-bound decode, c3-standard-22

10.5.1 Runtime memory: weights, KV cache, and compute buffer

The section shows how to calculate and provision CPU runtime memory from model weights, the KV cache, and the compute buffer rather than relying on GGUF file size alone.

Key terms: GGUF, weights, KV cache, compute buffer, mmap, FP16, decoder layers, KV heads, head dimension, ctx

10.5.2 Thread count and the cloud-CPU exception

This section uses a cloud-CPU thread sweep to show why SMT can improve decode throughput despite a physical-core-only heuristic, and how to choose a thread count for deployment.

Key terms: SMT, physical-core count, vCPU count, DRAM bandwidth, Q4_K_M decode, t=12 dip, SMT sibling, L1 and L2 caches, per-core sharing penalty, -t $(nproc)

10.6 Summary

Concepts explained in this chapter

← Chapter 9: Shipping with the Right Toolchain Chapter 11: Targeting Edge and Mobile Devices →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.