Chapter 10: Running on CPUs and Using GGUF
This introduction distinguishes CPU-first GGUF inference from graph-based deployment and explains why silent throughput losses matter in production.
After this chapter you will be able to
- GGUF deployment path. Distinguish CPU-first LLM inference with GGUF and llama.cpp from ONNX-based deployment. Explain why a GGUF file is not interchangeable with an ONNX graph.
- Silent CPU throughput losses. Explain how CPU inference choices can silently reduce throughput and why that loss matters to a production fleet. Describe the role of measurement on the target hardware without prescribing a benchmarking procedure.
- Bandwidth Bound CPU Decode. Explain why CPU LLM decode is bandwidth-bound and why the weight layout matters to CPU deployment.
- Utilization Dependent Token Cost. Apply the section's cost-per-million-tokens calculation and explain why the CPU–GPU cost gap changes with utilization.
- Prefill Versus Streaming Latency. Distinguish prompt prefill from streaming decode when judging whether CPU latency is acceptable.
- Concurrency And Batching Advantage. Explain how per-instance concurrency changes the CPU–GPU throughput and cost comparison.
- Operational Fit Deployment Tradeoff. Justify when CPU deployment may be worth a per-token premium, and identify workloads for which that case fails.
- GGUF as a self-contained container. Explain that GGUF is llama.cpp’s on-disk container for GGML and that one file holds the model architecture, tokenizer, and weights, with quantized tensors packed in fixed blocks for inference kernels.
- Legacy Block Scaling. Explain how a legacy quant stores weights in small blocks and why its single scale can hurt four-bit quality.
- K-Quant Scale Hierarchy. Describe the two levels of scaling in K-quants and why they improve quality relative to legacy quants at comparable or lower bits per weight.
- IQ-Quant Group Codebooks. Explain how IQ-quants encode groups of weights with codebook indices rather than assigning each weight its own integer code, and why this permits very low bit budgets.
- IQ-Quant Runtime Trade-Off. Explain why aggressive IQ-quants can decode more slowly than similarly sized K-quants on CPUs, while distinguishing the more nuanced four-bit case.
- GGUF File Section Layout. Describe the four GGUF sections in order and explain why packed tensor data dominates the file size.
- Memory-Mapped Weight Access. Explain how mmap lets the runtime access GGUF weights without first copying the whole file into a separate buffer.
- Sequential Decode And RAM Fit. Trace why sequential decode reads work well with mmap and why throughput falls if the weight file cannot stay in RAM.
- Self-describing GGUF metadata. Explain how GGUF metadata lets one file carry information that a typical Hugging Face checkpoint keeps across multiple artifacts. Describe the kinds of information stored and how they are represented.
- Per-tensor quantization recipe. Explain why a file labeled Q4_K_M contains Q4_K, Q6_K, and F32 tensors, and how the runtime uses their recorded types.
- Q8_0 Block Reconstruction. Describe the Q8_0 block layout and explain how one of its stored integers becomes a weight.
- Q6_K Superblock Reconstruction. Trace how Q6_K combines packed bits, a signed code, and hierarchical scales to reconstruct a weight.
- HF to GGUF pipeline. Trace how safetensors shards become an F16 GGUF and then a quantized artifact. Describe which work uses PyTorch and which work uses C++ binaries.
- Shared F16 Export Workflow. Trace how the model is exported once and then quantized into four production variants, including why norm and RMS tensors remain F32.
- Quantization Conversion Cost Differences. Explain why Q8_0, the two K-quants, and IQ4_XS take different amounts of time to produce, and distinguish this one-time conversion cost from runtime performance.
- Imatrix calibration and purpose. Explain how llama-imatrix uses a small corpus and an F16 model to produce calibration data for IQ4_XS, and how llama-quantize uses that data.
- GGUF Artifact Verification. Describe how to check that each GGUF file declares the format requested during conversion, and name the four axes used to evaluate the resulting variants.
- Size and Perplexity Trade-off. Explain what the Q4_K_M and IQ4_XS comparison shows about size and perplexity, and why the short perplexity run should not be treated as definitive for small differences.
- Fluency Beyond Perplexity. Explain why the section inspects deterministic generated text in addition to perplexity, and describe what the five variants' completions reveal.
- Weight Size and Decode Time. Explain why reducing bytes per weight tends to reduce decode time per token in this comparison.
- Hardware-Dependent Quant Choice. Distinguish k-quant arithmetic reconstruction from IQ-quant codebook lookups, and justify when Q4_K_M or IQ4_XS is the safer deployment choice.
- Hardware Dependent Format Rankings. Explain why the quantization-format ranking observed on Apple Silicon cannot be assumed to hold on x86. IQ4_XS uses codebook-lookup decoding, whose performance depends on instructions that vary across x86 generations.
- Cumulative x86 Kernel Capabilities. Describe how the relevant x86 ISA capabilities progress from AVX-2 and AVX-512 through VNNI and AMX, and how llama.cpp names a kernel family for the highest ISA its build targets.
- Compiled ISA versus CPU support. Explain how llama.cpp build-time ISA flags determine which kernels a binary contains, regardless of the host CPU’s capabilities. Describe how the three cumulative builds isolate AVX2, AVX-512 with VNNI, and AMX-INT8 on the same CPU.
- Q4_K dot product stages. Trace the four functional stages of the Q4_K kernel in order: scale broadcast, nibble unpack, fused dot, and horizontal reduce.
- Efficient loads and bandwidth. Explain why efficient kernel consumption matters for Q4_K_M's compact memory profile, and identify the bottleneck the section attributes to this inner loop.
- Decode bandwidth ceiling. Explain why reading roughly the full weight footprint for each generated token makes DRAM bandwidth a ceiling on decode throughput. Describe how the nearly identical F16 results across builds serve as a baseline for that explanation.
- Limited AVX-512 decode gains. Explain why AVX-512 and VNNI yield only small gains over AVX-2 for the four integer formats in single-token decode, despite their faster dot-product capability. Distinguish this workload from prompt processing.
- AMX tile overhead in decode. Explain why the AMX-enabled build is slower than plain AVX-512 on every tested quantized format during batch-1 decode, even though AMX helps prompt processing.
- IQ4_XS Throughput Lead. State how IQ4_XS decode throughput compares with Q4_K_M on the measured kernel families, alongside its previously established size and perplexity advantage.
- Format Versus Kernel Effect. Distinguish the effect of kernel-family choice on the IQ4_XS-to-Q4_K_M ratio from the larger throughput effect of format choice.
- Decode and prompt build choices. Explain why the measured Sapphire Rapids deployment favors AVX-512 with VNNI and AMX off for single-stream decode, while longer prompts can favor AMX. Describe the choice between routing requests to two binaries and accepting a single-binary trade-off.
- Default format and fallback. Justify IQ4_XS as the default format for the measured CPU decode kernels and identify when Q4_K_M is the safer production fallback.
- Runtime RAM versus file size. Explain why the quantized model's disk size is not a safe RAM provisioning target for CPU inference.
- Physical cores and thread count. Describe the documented physical-core thread recommendation, its reasoning, and the cloud-CPU measurement that challenges it.
- Three Runtime Memory Terms. Describe the three parts of a loaded model’s memory footprint and how each changes: weights are fixed by the GGUF file, the KV cache grows with context, and the compute buffer depends on vocabulary size and embedding width rather than context.
- FP16 KV Cache Calculation. Apply the section’s formula to calculate FP16 KV cache size from decoder layers, KV heads, head dimension, and context length, and explain the two factors of two.
- GQA KV Head Savings. Distinguish MHA from GQA in terms of KV heads and explain why the Llama-3-8B example uses one-quarter the KV cache of Llama-2-7B at the same context.
- Runtime Memory Provisioning. Explain why provisioning against GGUF file size can cause an out-of-memory failure and describe how the section obtains each term of a runtime memory estimate.
- Cloud SMT Throughput Gain. Explain why Q4_K_M decode on the 22-vCPU cloud VM runs faster at 22 threads than at 11 physical-core threads, despite the usual bandwidth-saturation heuristic.
- Cloud Thread Tuning. Apply the section's deployment recommendation for thread count and affinity to single-stream Q4_K_M decode on Sapphire Rapids cloud CPUs.
What it builds on
- Choose graphs, runtimes, and artifacts for deployment (chapter 9) · Introduces the distinction between deployment graphs, runtimes, and artifacts.
- Use group and block schemes and understand memory layout (chapter 3) · Quantizing fixed-size groups of weights.
- Package models for serving and distribution (chapter 9) · The separate manifest-sidecar packaging pattern used with ORT and TensorRT.
- Build TensorRT engines with real integer execution (chapter 9) · The earlier sensitivity-driven pattern of excluding selected nodes from quantization.
- Quantize weights with per-tensor and per-channel choices (chapter 3) · Uses the earlier idea of quantized weights and their scales.
- Quantize transformer key-value cache with mixed precision (chapter 3) · The transformer KV cache as a memory cost of inference.
Section by section
Introduction
This introduction distinguishes CPU-first GGUF inference from graph-based deployment and explains why silent throughput losses matter in production.
- GGUF deployment path: Distinguish CPU-first LLM inference with GGUF and llama.cpp from ONNX-based deployment. Explain why a GGUF file is not interchangeable with an ONNX graph.
Why it matters: It explains why CPU-first GGUF deployment needs a different artifact and runtime path from ONNX deployment. - Silent CPU throughput losses: Explain how CPU inference choices can silently reduce throughput and why that loss matters to a production fleet. Describe the role of measurement on the target hardware without prescribing a benchmarking procedure.
Why it matters: A configuration that runs without errors can still increase serving cost or cause tail-latency failures.
Key terms: GGUF, K-quant, IQ-quant, operator-level INT8 dispatch, single-binary C++ inference engine, throughput, AMX build flag, batch-1 decode, thread count, CPU replica
10.1 Know when CPU wins on cost and scale-out
This section compares CPU and GPU LLM inference by showing how utilization, latency, concurrency, memory capacity, and operational fit change the deployment decision.
- Bandwidth Bound CPU Decode: Explain why CPU LLM decode is bandwidth-bound and why the weight layout matters to CPU deployment.
Why it matters: It explains why a CPU deployment depends on more than processor speed. - Utilization Dependent Token Cost: Apply the section's cost-per-million-tokens calculation and explain why the CPU–GPU cost gap changes with utilization.
Why it matters: A saturated benchmark can misrepresent what a deployment will pay under its actual demand. - Prefill Versus Streaming Latency: Distinguish prompt prefill from streaming decode when judging whether CPU latency is acceptable.
Why it matters: A readable output stream does not compensate for a long wait before the first token. - Concurrency And Batching Advantage: Explain how per-instance concurrency changes the CPU–GPU throughput and cost comparison.
Why it matters: Single-stream results can understate the GPU's lead in a busy shared service. - System RAM Model Headroom: Describe when CPU system memory makes a model runnable despite slow decode.
Why it matters: Capacity can make CPU inference viable when the compared GPU cannot hold the model. - Operational Fit Deployment Tradeoff: Justify when CPU deployment may be worth a per-token premium, and identify workloads for which that case fails.
Why it matters: The cheapest saturated token is not always the best way to serve an actual workload.
Key terms: bandwidth-bound, decode, weight matrix, byte layout, dollars per million generated tokens, utilization, TPS ceiling, replicas, prefill, latency-to-first-token
10.2 Understand the GGUF format and its variants
Introduces GGUF as a self-contained model file whose quantized tensor blocks can be consumed directly by llama.cpp inference kernels.
- GGUF as a self-contained container: Explain that GGUF is llama.cpp’s on-disk container for GGML and that one file holds the model architecture, tokenizer, and weights, with quantized tensors packed in fixed blocks for inference kernels.
Why it matters: This explains why a GGUF artifact can contain both the model information and the packed weights used by the runtime.
Key terms: GGUF, GGML, llama.cpp, quantized tensor, fixed block layout
10.2.1 Three quant families, three generations
This subsection explains how legacy, K-quant, and IQ-quant GGUF formats compress weights differently, and the resulting quality and CPU-speed trade-offs.
- Legacy Block Scaling: Explain how a legacy quant stores weights in small blocks and why its single scale can hurt four-bit quality.
Why it matters: This explains the quality limitation that the newer families address. - K-Quant Scale Hierarchy: Describe the two levels of scaling in K-quants and why they improve quality relative to legacy quants at comparable or lower bits per weight.
Why it matters: The scale hierarchy explains why bit count alone does not predict quantized-model quality. - K-Quant Recipe Variants: Distinguish the uniform S recipe from M and L recipes that promote selected sensitive tensors to higher precision.
Why it matters: A format name can describe a mixed-precision recipe rather than one precision applied everywhere. - IQ-Quant Group Codebooks: Explain how IQ-quants encode groups of weights with codebook indices rather than assigning each weight its own integer code, and why this permits very low bit budgets.
Why it matters: Groupwise codebook encoding is the fundamental difference between IQ-quants and the earlier families. - Importance Matrix Fitting: Describe how a calibration-derived importance matrix guides IQ-quant codebook fitting and why it is especially important below four bits.
Why it matters: The importance matrix helps preserve output quality when the available bit budget is especially tight. - IQ-Quant Runtime Trade-Off: Explain why aggressive IQ-quants can decode more slowly than similarly sized K-quants on CPUs, while distinguishing the more nuanced four-bit case.
Why it matters: The smallest artifact is not necessarily the fastest choice for CPU inference.
Key terms: legacy quants, Q4_0, Q8_0, FP16 scale factor, k-quants, superblock, super-scale, sub-scales, bits per weight, S variant
10.2.2 Four sections of the GGUF file
Explains the GGUF file layout and how memory mapping, sequential weight access, and alignment support CPU inference.
- GGUF File Section Layout: Describe the four GGUF sections in order and explain why packed tensor data dominates the file size.
Why it matters: The layout shows where the weights live and why accessing them efficiently matters most. - Little-Endian Header Reading: Explain how the section reads multi-byte header integers and identify what the illustrated header fields count.
Why it matters: Byte order makes the header values in a hex dump interpretable. - Memory-Mapped Weight Access: Explain how mmap lets the runtime access GGUF weights without first copying the whole file into a separate buffer.
Why it matters: This explains how the runtime can start using a large weight file without first duplicating it in RAM. - Sequential Decode And RAM Fit: Trace why sequential decode reads work well with mmap and why throughput falls if the weight file cannot stay in RAM.
Why it matters: It explains both the fast steady-state case and the sharp penalty when weights exceed available RAM. - Aligned Tensor Data Boundary: Explain why GGUF starts tensor data on a 32-byte file boundary and what the illustrated padding accomplishes.
Why it matters: The boundary connects the file layout to efficient CPU weight reads.
Key terms: header, metadata key-value section, tensor descriptor table, packed tensor data, little-endian, u32, u64, GGUF version, tensor count, metadata key-value count
10.2.3 Metadata makes the file self-describing
This section explains how embedded GGUF metadata makes a model file self-describing, while limiting which runtimes can serve it.
- Self-describing GGUF metadata: Explain how GGUF metadata lets one file carry information that a typical Hugging Face checkpoint keeps across multiple artifacts. Describe the kinds of information stored and how they are represented.
Why it matters: Embedded metadata makes a GGUF artifact easier to keep together and inspect. - Metadata and runtime limits: Distinguish GGUF metadata from the attention KV cache, and explain the deployment tradeoff of embedding recipe metadata rather than keeping it in a sidecar file.
Why it matters: A self-contained file does not make the format interchangeable with other serving runtimes.
Key terms: metadata section, typed key-value pairs, namespaced names, quantization recipe label, attention KV cache, manifest-sidecar pattern, format lock-in, re-export
10.2.4 The packing is not uniform
This section explains how a GGUF file records each tensor’s format and why a Q4_K_M file contains a mix of quantization types.
- Tensor descriptor contents: Describe where the tensor descriptor table begins and what each descriptor records.
Why it matters: The descriptors show where a tensor’s format and packed data are recorded. - Per-tensor quantization recipe: Explain why a file labeled Q4_K_M contains Q4_K, Q6_K, and F32 tensors, and how the runtime uses their recorded types.
Why it matters: A GGUF filename alone does not tell you the format of every tensor inside it.
Key terms: tensor descriptor table, ggml type code, byte offset, per-tensor type codes, Q4_K_M, Q4_K, Q6_K, F32
10.2.5 Inside one block: from bytes to a weight
This section shows how raw bytes in Q8_0 and Q6_K GGUF blocks become reconstructed weights, and why a GGUF recipe need not use one format throughout the file.
- Q8_0 Block Reconstruction: Describe the Q8_0 block layout and explain how one of its stored integers becomes a weight.
Why it matters: This is the simplest example of turning GGUF block bytes back into weights. - Q6_K Superblock Reconstruction: Trace how Q6_K combines packed bits, a signed code, and hierarchical scales to reconstruct a weight.
Why it matters: It makes the extra packing and scale hierarchy of k-quants concrete rather than treating Q6_K as just a format name. - Mixed Formats Within Recipes: Explain why a Q4_K_M GGUF file can contain a tensor stored as Q6_K.
Why it matters: The file's recipe name alone does not tell you how to decode every tensor block.
Key terms: Q8_0, FP16 scale, signed INT8s, Q6_K, superblock, ql, qh, sub-scale, super-scale, Q4_K_M
10.3 Convert large language models to CPU-friendly artifacts
Introduces the GGUF conversion pipeline and identifies the single hardware and software run behind the results that follow.
10.3.1 The pipeline at a glance
This section maps the Hugging Face-to-GGUF conversion pipeline to its Python and C++ tools and identifies three first-run pitfalls.
- HF to GGUF pipeline: Trace how safetensors shards become an F16 GGUF and then a quantized artifact. Describe which work uses PyTorch and which work uses C++ binaries.
Why it matters: The division of work tells you which artifact each tool consumes and what each stage is responsible for. - Gated model download access: Explain what access is required before downloading the section’s Llama-2 model and what an HTTP 401 or 403 indicates at that stage.
Why it matters: Resolving gated access before the run prevents a download failure from blocking conversion. - Clean conversion environment: Justify using a fresh Python 3.13 virtual environment with the cloned repo’s conversion requirements for the GGUF converter.
Why it matters: An isolated conversion environment avoids a runtime conflict that can interrupt the export. - One shot verification binary: Distinguish llama-completion from llama-cli for one-shot verification and describe the build implication.
Why it matters: Choosing and building the intended binary prevents an apparent hang during generation checks.
Key terms: safetensors shards, convert_hf_to_gguf.py, F16 GGUF, llama-quantize, llama-imatrix, llama-completion, meta-llama/Llama-2-7b-hf, community license, huggingface-cli login, gate access
10.3.2 F16 export and the four quantization runs
This subsection explains how one F16 GGUF export feeds four quantization runs and why their one-time conversion costs differ.
- Shared F16 Export Workflow: Trace how the model is exported once and then quantized into four production variants, including why norm and RMS tensors remain F32.
Why it matters: A single export supplies all four variants without repeating the PyTorch conversion. - Quantization Conversion Cost Differences: Explain why Q8_0, the two K-quants, and IQ4_XS take different amounts of time to produce, and distinguish this one-time conversion cost from runtime performance.
Why it matters: Conversion time reflects work done to create an artifact, not a cost paid on every inference.
Key terms: convert_hf_to_gguf.py, safetensors index, GGUF, F16, F32, llama-quantize, Q8_0, Q5_K_M, Q4_K_M, Q6_K
10.3.3 The imatrix is the IQ-quant prerequisite
This subsection explains how an imatrix calibrates IQ4_XS quantization and why its tensor-upgrade choices produce a smaller file than Q4_K_M.
- Imatrix calibration and purpose: Explain how llama-imatrix uses a small corpus and an F16 model to produce calibration data for IQ4_XS, and how llama-quantize uses that data.
Why it matters: It explains the extra calibration step in the IQ4_XS conversion without mistaking it for model training. - Four-bit tensor upgrade choices: Distinguish the Q6_K tensor upgrades used by Q4_K_M and IQ4_XS in this run, and relate their different upgrade footprints to file size.
Why it matters: A shared nominal bit width does not imply the same mix of tensor formats or the same artifact size.
Key terms: imatrix, llama-imatrix, llama-quantize, F16 model, sidecar GGUF, per-tensor activation magnitudes, IQ4_XS, Q4_K_M, Q6_K, upgrade footprint
10.3.4 Verify size, perplexity, fluency, and decode throughput
This section verifies five GGUF variants and uses size, perplexity, generated text, and decode speed to weigh their deployment trade-offs.
- GGUF Artifact Verification: Describe how to check that each GGUF file declares the format requested during conversion, and name the four axes used to evaluate the resulting variants.
Why it matters: A benchmark comparison is only trustworthy if every file is actually the format its label claims. - Size and Perplexity Trade-off: Explain what the Q4_K_M and IQ4_XS comparison shows about size and perplexity, and why the short perplexity run should not be treated as definitive for small differences.
Why it matters: A smaller artifact can retain strong measured quality, but close scores need enough evaluation data to support a claim. - Fluency Beyond Perplexity: Explain why the section inspects deterministic generated text in addition to perplexity, and describe what the five variants' completions reveal.
Why it matters: Reading actual completions can catch changes that a single quality score conceals. - Weight Size and Decode Time: Explain why reducing bytes per weight tends to reduce decode time per token in this comparison.
Why it matters: The speed benefit of quantization follows from moving fewer weight bytes for every generated token. - Hardware-Dependent Quant Choice: Distinguish k-quant arithmetic reconstruction from IQ-quant codebook lookups, and justify when Q4_K_M or IQ4_XS is the safer deployment choice.
Why it matters: The smallest file is not automatically the fastest choice on the hardware where it will run.
Key terms: gguf.GGUFReader, general.file_type, LlamaFileType, size, perplexity, fluency, IQ4_XS, Q4_K_M, imatrix, standard-error band
10.4 Compare kernel families and what they imply
This section explains why quantization-format rankings may change across x86 kernel families and sets up a controlled comparison of those families.
- Hardware Dependent Format Rankings: Explain why the quantization-format ranking observed on Apple Silicon cannot be assumed to hold on x86. IQ4_XS uses codebook-lookup decoding, whose performance depends on instructions that vary across x86 generations.
Why it matters: It keeps a hardware-specific benchmark ranking from becoming a blanket deployment recommendation. - Cumulative x86 Kernel Capabilities: Describe how the relevant x86 ISA capabilities progress from AVX-2 and AVX-512 through VNNI and AMX, and how llama.cpp names a kernel family for the highest ISA its build targets.
Why it matters: These capabilities explain why builds targeting different ISAs can execute quantized workloads differently. - Controlled Kernel Comparison: Describe what the companion benchmark holds fixed and what it varies when comparing GGUF formats across llama.cpp builds.
Why it matters: Holding the hardware and artifacts fixed makes the compiled ISA the intended difference between kernel-build comparisons.
Key terms: IQ4_XS, Q4_K_M, codebook-lookup decode, x86, ISA, AVX-2, AVX-512, VNNI, AMX, kernel families
10.4.1 Three builds, one CPU
This subsection shows how three llama.cpp builds on the same CPU expose different instruction kernels, and why AMX execution also depends on matrix size.
- Compiled ISA versus CPU support: Explain how llama.cpp build-time ISA flags determine which kernels a binary contains, regardless of the host CPU’s capabilities. Describe how the three cumulative builds isolate AVX2, AVX-512 with VNNI, and AMX-INT8 on the same CPU.
Why it matters: It prevents you from attributing a missing instruction path to the CPU when it was excluded from the binary. - Separate AMX dispatch path: Distinguish AMX from the AVX integer-kernel paths and explain when an AMX-enabled build uses AMX rather than falling back to AVX-512.
Why it matters: Including an AMX kernel in a build does not mean every matrix multiplication will run on AMX.
Key terms: build-time ISA flags, GGML_AVX2, GGML_AVX512, GGML_AMX_TILE, build-avx2, build-avx512, AMX, AMX-INT8, AVX-512, tile-load overhead
10.4.2 Inside one Q4_K dot product
This section traces the stages of a Q4_K dot-product kernel and explains how its memory access and instruction choices affect CPU inference.
- Q4_K dot product stages: Trace the four functional stages of the Q4_K kernel in order: scale broadcast, nibble unpack, fused dot, and horizontal reduce.
Why it matters: These stages show what the CPU must do with a packed Q4_K superblock, beyond storing it compactly. - Efficient loads and bandwidth: Explain why efficient kernel consumption matters for Q4_K_M's compact memory profile, and identify the bottleneck the section attributes to this inner loop.
Why it matters: A compact weight format helps only if its kernel can use the loaded data efficiently. - IQ4_XS lookup instruction differences: Distinguish IQ4_XS's codebook lookup from Q4_K's nibble unpack, and describe how that lookup differs between AVX-512 with VBMI and AVX-2.
Why it matters: The same quantized format can require different instruction sequences on different CPU builds.
Key terms: scale broadcast, nibble unpack, fused dot, horizontal reduce, superblock, Q4_K_M, scatter-gathers, DRAM-to-L2 bandwidth, ALU rate, IQ4_XS
10.4.3 Decode throughput across the matrix
This section compares single-token decode throughput across quantization formats and CPU builds, explaining why memory bandwidth limits ISA gains and why AMX can slow decode.
- Decode bandwidth ceiling: Explain why reading roughly the full weight footprint for each generated token makes DRAM bandwidth a ceiling on decode throughput. Describe how the nearly identical F16 results across builds serve as a baseline for that explanation.
Why it matters: It explains why a faster CPU instruction does not necessarily produce faster token generation. - Limited AVX-512 decode gains: Explain why AVX-512 and VNNI yield only small gains over AVX-2 for the four integer formats in single-token decode, despite their faster dot-product capability. Distinguish this workload from prompt processing.
Why it matters: It keeps prompt-processing speedups from being mistaken for expected decode speedups. - AMX tile overhead in decode: Explain why the AMX-enabled build is slower than plain AVX-512 on every tested quantized format during batch-1 decode, even though AMX helps prompt processing.
Why it matters: It shows why enabling a theoretically faster kernel path can regress token generation. - Q5_K_M regression gap: Explain the section's likely reason Q5_K_M loses more throughput with AMX than the other tested formats, while distinguishing that explanation from the tile overhead shared by all formats.
Why it matters: It explains why the same AMX build penalty is particularly severe for one format without treating the proposed kernel-level cause as certain.
Key terms: decode, bandwidth-bound, weight footprint, F16, AVX-2, AVX-512, VNNI, single-token decode, prompt processing, AMX
10.4.4 The IQ4_XS / Q4_K_M crossover
This section compares IQ4_XS and Q4_K_M decode throughput across CPU kernel families, explains why their speed gap narrows with AMX, and puts that effect in perspective.
- IQ4_XS Throughput Lead: State how IQ4_XS decode throughput compares with Q4_K_M on the measured kernel families, alongside its previously established size and perplexity advantage.
Why it matters: A format choice needs to hold up on throughput as well as size and perplexity. - AMX Gap Narrowing: Explain why the IQ4_XS throughput lead narrows on the AMX-enabled build without reversing.
Why it matters: The work inside each format's inner loop affects how much it benefits from a CPU instruction family. - Format Versus Kernel Effect: Distinguish the effect of kernel-family choice on the IQ4_XS-to-Q4_K_M ratio from the larger throughput effect of format choice.
Why it matters: This comparison helps prioritize the format decision over a smaller change in the relative results across kernel families.
Key terms: IQ4_XS, Q4_K_M, decode throughput, AVX-2, AVX-512+VNNI, AVX-512+AMX, dense INT8 multiply-accumulate, codebook gather, TDPBSSD, kernel-family effect
10.4.5 What to ship
This section recommends CPU builds and quantization formats for Llama-2-7B deployment, distinguishing single-stream decode from prompt processing.
- Decode and prompt build choices: Explain why the measured Sapphire Rapids deployment favors AVX-512 with VNNI and AMX off for single-stream decode, while longer prompts can favor AMX. Describe the choice between routing requests to two binaries and accepting a single-binary trade-off.
Why it matters: A build optimized for generating one token at a time may not be the best build for processing a long prompt. - Default format and fallback: Justify IQ4_XS as the default format for the measured CPU decode kernels and identify when Q4_K_M is the safer production fallback.
Why it matters: The fastest measured format is useful only when the target runtime can support it effectively. - Compiled kernels and speed limits: Explain how build-time flags determine which llama.cpp integer kernels are available and why richer x86 instructions yield only modest gains for the measured decode workload.
Why it matters: CPU capabilities alone do not guarantee that a deployed binary contains or benefits greatly from the corresponding kernels.
Key terms: single-stream decode, prompt processing, AVX-512, VNNI, AMX, IQ4_XS, Q4_K_M, 16-entry codebooks, decode throughput, build-time CMake flags
10.5 Set realistic expectations for throughput and memory
This section challenges two CPU deployment assumptions: that model file size determines RAM requirements and that one thread per physical core always gives the best decode throughput.
- Runtime RAM versus file size: Explain why the quantized model's disk size is not a safe RAM provisioning target for CPU inference.
Why it matters: Provisioning from file size alone can leave a CPU deployment without enough memory. - Physical cores and thread count: Describe the documented physical-core thread recommendation, its reasoning, and the cloud-CPU measurement that challenges it.
Why it matters: A physical-core rule of thumb may give a misleading throughput expectation on a cloud CPU.
Key terms: Q4_K_M, RAM, OOM, -t, physical core, Simultaneous Multi-Threading (SMT), bandwidth-bound decode, c3-standard-22
10.5.1 Runtime memory: weights, KV cache, and compute buffer
The section shows how to calculate and provision CPU runtime memory from model weights, the KV cache, and the compute buffer rather than relying on GGUF file size alone.
- Three Runtime Memory Terms: Describe the three parts of a loaded model’s memory footprint and how each changes: weights are fixed by the GGUF file, the KV cache grows with context, and the compute buffer depends on vocabulary size and embedding width rather than context.
Why it matters: A model that fits on disk may need substantially more memory while serving requests. - FP16 KV Cache Calculation: Apply the section’s formula to calculate FP16 KV cache size from decoder layers, KV heads, head dimension, and context length, and explain the two factors of two.
Why it matters: The cache can be calculated from architecture and context before choosing a machine. - GQA KV Head Savings: Distinguish MHA from GQA in terms of KV heads and explain why the Llama-3-8B example uses one-quarter the KV cache of Llama-2-7B at the same context.
Why it matters: Attention architecture can change long-context memory requirements even when model sizes are similar. - Runtime Memory Provisioning: Explain why provisioning against GGUF file size can cause an out-of-memory failure and describe how the section obtains each term of a runtime memory estimate.
Why it matters: Sizing for the complete runtime footprint prevents a model from loading successfully but failing on a long request.
Key terms: GGUF, weights, KV cache, compute buffer, mmap, FP16, decoder layers, KV heads, head dimension, ctx
10.5.2 Thread count and the cloud-CPU exception
This section uses a cloud-CPU thread sweep to show why SMT can improve decode throughput despite a physical-core-only heuristic, and how to choose a thread count for deployment.
- Cloud SMT Throughput Gain: Explain why Q4_K_M decode on the 22-vCPU cloud VM runs faster at 22 threads than at 11 physical-core threads, despite the usual bandwidth-saturation heuristic.
Why it matters: A physical-core limit can leave useful CPU capacity untapped on a cloud VM. - Twelve Thread Throughput Dip: Explain the dip when the sweep moves from 11 to 12 threads and why throughput recovers as more SMT siblings are used.
Why it matters: A one-thread setback need not mean that all higher thread counts will be slower. - Cloud Thread Tuning: Apply the section's deployment recommendation for thread count and affinity to single-stream Q4_K_M decode on Sapphire Rapids cloud CPUs.
Why it matters: Measuring the actual deployment configuration avoids both an overly restrictive thread limit and harmful affinity settings. - Per VM Benchmark Variability: Distinguish the repeatable shape of the thread-scaling curve from absolute throughput numbers that vary across shared cloud VMs.
Why it matters: It keeps you from treating a single shared-VM tokens-per-second result as a portable performance guarantee.
Key terms: SMT, physical-core count, vCPU count, DRAM bandwidth, Q4_K_M decode, t=12 dip, SMT sibling, L1 and L2 caches, per-core sharing penalty, -t $(nproc)
10.6 Summary
Concepts explained in this chapter
- throughputThe amount of computation or output completed per unit of time, including its retention during sustained operation.
- decode phaseDecode is autoregressive inference that generates output tokens one at a time using the preceding context.
- AMX-INT8Intel Advanced Matrix Extensions (AMX) are processor extensions that accelerate matrix operations using tile registers and dedicated instructions.
- AVX-512AVX-512 is a family of x86 instruction-set extensions providing 512-bit SIMD operations and related instruction subsets.
- IQ-quantizationIQ-quant is a family of llama.cpp importance-aware, low-bit quantization schemes that use non-uniform representations to compress model weights.
- inferenceInference is the execution of a trained machine-learning model to produce outputs from inputs.
- superblockA superblock is a group of weights in a K-quant scheme that has an overall scale and contains smaller groups with their own scales.
- AVX2AVX2 is an x86 SIMD instruction-set extension that adds 256-bit integer vector operations to AVX.
- K-quantizationK-quant is a family of llama.cpp low-bit quantization schemes that store model values in structured blocks with associated scaling information.
- q4_K_M formatQ4_K_M is a llama.cpp quantization format that stores model weights primarily using four-bit K-quant blocks.
- IQ4_XSIQ4_XS is a llama.cpp importance-aware quantization format that represents model values using four-bit indices and a codebook.
- concurrencyConcurrency is the number of requests or users being served simultaneously by a model instance.
- memory bandwidthMemory bandwidth is the rate at which data can be transferred between memory and computational components.
- prefillPrefill is the initial computation that processes a language-model prompt before token generation begins.
- resource utilizationUtilization is the fraction of an instance's available capacity that is actively used over a period.
- time to first tokenThe elapsed time from submitting a language-model request until its first generated token is available.
- thread countThread count is the configured number of concurrent execution threads assigned to a computation or runtime.
- AVX-512 VNNIAVX-512 VNNI is an x86 instruction-set extension that adds low-precision integer dot-product operations to AVX-512.
- q8_0Q8_0 is a block quantization format that stores 32 weights in 34 bytes using an FP16 scale and 32 signed INT8 values.
- GGUFGGUF is a file format for packaging quantized machine-learning models and their associated metadata for inference runtimes.
- simultaneous multithreadingSMT is a processor capability that allows multiple hardware threads to share the resources of one physical core.
- imatrixAn imatrix is a collection of per-tensor activation-magnitude estimates used to weight quantization toward tensors important to model loss.
- processor affinityCPU affinity is a policy or mechanism that restricts a process or thread to specified processor units.
- thread-count sweepA thread-count sweep measures a workload across multiple configured thread counts to identify a suitable setting.
- perplexityPerplexity is a language-model metric measuring how well a probability model predicts a sequence, with lower values indicating better predictive performance.
- CPUA CPU is a general-purpose processor that executes program instructions and arithmetic operations.
- key-value cacheRuntime storage for attention keys and values from prior tokens, allowing an autoregressive transformer to reuse them when processing later tokens.
- Q6_KQ6_K is a k-quant format that stores 256 weights per superblock using 6-bit codes and hierarchical scales.
- bits per weightBits per weight is the average number of bits used to represent each model weight in a quantized format.
- importance matrixNumerical data representing the relative importance of model weights for guiding quantization.
- sub-scaleA sub-scale is a quantization scale applied to a smaller chunk of weights within a hierarchical quantization scheme.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
