How deeply does LLMs in Production cover quantization?

Defines the precision trade-off and works through PyTorch and llama.cpp procedures, without deriving the underlying numerical mathematics.

Quantization

As explained in LLMs in Production

Quantization is the conversion of model parameters or computations from higher-precision numerical representations to lower-precision representations.

A model that runs comfortably during training may be too large to serve. Quantization reduces numerical precision to lower memory requirements, making compression a practical first step when disk space, RAM, GPU memory, or serving cost is the constraint. Less precision is not free: stronger compression can raise perplexity and make an LLM’s behavior less stable or predictable. section 3.3.1 section 9.3.1 section 11.4.2

Think of reporting a ruler measurement to far more decimal places than the ruler can verify. Keeping every digit does not necessarily make the measurement more useful. Model numbers pose a harder version of that question: reducing precision saves space, but reducing it too far increases error. The useful setting is therefore the one that meets the hardware limit while preserving acceptable performance—not simply the fewest bits available. section 3.3.1 section 11.4.2

Choose a route based on what you are deploying:

For a trained PyTorch model, quantize after training and compare the resulting artifact with the original. A dynamic INT8 example applies configurations to embedding and linear layers and reports a drop from about 716 MB to 18 MB on disk; check model behavior as well as size. section 9.3.1
For a llama.cpp edge deployment, convert the model to a GGUF checkpoint, then quantize it—or download an already-quantized GGUF file. A 7B model in Q4_K_M is about 4.37 GB on disk and requires up to 6.87 GB RAM in the comparison table. section 11.2 section 11.4.2
If the model still will not fit, compare a smaller model with a lower-bit format. Q2_K is smaller, but carries significant quality loss and is not recommended for most purposes. Account for VRAM as well as RAM when using a GPU. section 11.4.2

Listing 9.5 Quantization

llama.to("cpu")
qconfig_dict = {
    torch.nn.Embedding: torch.quantization.float_qparams_weight_only_qconfig,
    torch.nn.Linear: torch.quantization.default_dynamic_qconfig,
}
dynamic_quantized_llama = torch.quantization.quantize_dynamic(    #1
    llama, qconfig_dict, dtype=torch.qint8
)
#SimpleLlama size: 716.504MB
#SimpleLlama size: 18.000MB

The PyTorch listing isolates the dynamic INT8 call and shows the before-and-after disk sizes for a trained model. section 9.3.1

Table 11.1 Comparison of key attributes for different llama.cpp quantization methods for a 7B parameter model

Quant method	Bits	Size (GB)	Max RAM required (GB)	Use case	Params (billions)
`Q2_K`	2	2.72	5.22	Significant quality loss; not recommended for most purposes	7
`Q3_K_S`	3	3.16	5.66	Very small, high loss of quality	7
`Q3_K_M`	3	3.52	6.02	Very small, high loss of quality	7
`Q3_K_L`	3	3.82	6.32	Small, substantial quality loss	7
`Q4_0`	4	4.11	6.61	Legacy; small, very high loss of quality; prefer using `Q3_K_M`	7
`Q4_K_S`	4	4.14	6.64	Small, greater quality loss	7
`Q4_K_M`	4	4.37	6.87	Medium, balanced quality; recommended	7
`Q5_0`	5	5.00	7.50	Legacy; medium, balanced quality; prefer using `Q4_K_M`	7
`Q5_K_S`	5	5.00	7.50	Large, low loss of quality; recommended	7
`Q5_K_M`	5	5.13	7.63	large, very low loss of quality; recommended	7
`Q6_K`	6	5.94	8.44	Very large, extremely low loss of quality	7
`Q8_0`	8	7.70	10.20	Very large, extremely low loss of quality; not recommended	7

Use the 7B-format comparison to check bits, disk size, maximum RAM, and stated quality before choosing a llama.cpp format. section 11.4.2

Quantization can make inference feasible on constrained hardware, but fitting is not the same as serving well. An LLM on a Raspberry Pi may still be too slow for its intended use. Test output quality and inference speed in the target environment before treating a smaller checkpoint as production-ready. section 11.5 section 9.3.1

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Domain-Specific Small Language ModelsMethods that reduce numeric precision or representational size of model values to improve efficiency, especially for inference; here the absmax method is a specific quantization procedure.
liveBook manning.com

AI-Powered SearchQuantization is the transformation of numerical values into a smaller set of discrete values to reduce representation precision and resource requirements.
liveBook manning.com

Deep Learning with Python, Third EditionThe technique of reducing numeric precision of model weights or activations to speed up computation while preserving output behavior as much as possible.
liveBook manning.com

Deep Learning with R, Third EditionModel quantization is the conversion of model computations or parameters to lower-precision numerical representations to reduce resource requirements.
liveBook manning.com

Math and Architectures of Deep LearningThe conversion of floating-point region boundaries or ratios into integer-aligned indices for feature extraction.
liveBook manning.com

Generative AI in ActionQuantization is the reduction of numerical precision used to represent model parameters, typically to reduce memory use and computational cost.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA model optimization technique that reduces numerical precision of parameters and operations to lower memory use and often speed up inference.
liveBook manning.com

Transformers in ActionQuantization is the transformation of numerical model parameters or activations into lower-precision representations to reduce memory consumption or computational cost.
liveBook manning.com

The Complete Obsolete Guide to Generative AIA quantized model is a machine-learning model whose numerical parameters or activations are represented at reduced precision to lower resource requirements.
liveBook manning.com

Time Series Forecasting Using Foundation ModelsQuantization is the conversion of continuous numerical values into a finite set of discrete levels or tokens.
liveBook manning.com

Natural Language Processing in Action, Second EditionQuantization is the transformation of numerical values into lower-precision discrete representations, often to reduce memory use and accelerate computation.
liveBook manning.com

Building Reliable AI SystemsQuantization reduces the numerical precision used to represent model values, often lowering storage or computation requirements.

Compare how 13 books explain quantization →

Related concepts

QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
scalar quantizationincludes · Scalar quantization maps numerical scalar values to a finite set of discrete levels, typically reducing their precision or range.
QA-LoRAused by · QA-LoRA is a low-rank adaptation method that integrates quantization-aware operations and group-wise processing to support efficient low-bit adaptation.
post-training quantizationincludes · Post-training quantization is the conversion of a trained model’s parameters or computations to lower-precision representations without additional training.
quantization-aware trainingincludes · Quantization-aware training optimizes a model while simulating or incorporating reduced-precision effects so it adapts to quantized representations.
PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.