Quantization
Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
LLMs in Production
Quantization is the conversion of model parameters or computations from higher-precision numerical representations to lower-precision representations.
- How it is used
- Reduces model memory requirements to make LLM serving feasible on constrained hardware.
- How deep it goes
- Explains precision and deployment trade-offs, then demonstrates post-training quantization and format selection without deriving floating-point arithmetic.
- Examples
- Uses an INT8 PyTorch model and a Q4_K_M 7B GGUF model on a Raspberry Pi.
- What you can do after
- Apply it via PyTorch or llama.cpp and compare quantization formats against memory and quality constraints.
Domain-Specific Small Language Models
Methods that reduce numeric precision or representational size of model values to improve efficiency, especially for inference; here the absmax method is a specific quantization procedure.
- How it is used
- Reduces LLM inference memory requirements while targeting practical latency, throughput, and deployment constraints.
- How deep it goes
- Works through 8-bit scaling and reconstruction, then applies 4-bit libraries; defers GPTQ’s low-level mathematics to its implementation sources.
- Examples
- Uses GPT-2 absmax weight comparisons, GPT-2 XL AutoGPTQ configuration, and an 8-bit antibody-model latency benchmark.
- What you can do after
- Apply it via AutoGPTQ, ggml, or Bitsandbytes, then evaluate model size, inference latency, and output quality.
Read how Domain-Specific Small Language Models explains quantization →
CUDA for LLMs
Quantization represents numerical values using a limited set of discrete levels, often to reduce model storage and computation costs.
- How it is used
- Reduces model storage and memory traffic to make inference practical on constrained GPUs.
- How deep it goes
- Works through scaling, rounding, clamping, CUDA kernels, granularity, calibration, and deployment tradeoffs.
- Examples
- Uses a four-weight INT8 conversion and packed INT4 kernels to illustrate numerical mapping and storage.
- What you can do after
- Build quantization and dequantization kernels from scratch, then evaluate accuracy and profile inference overhead.
AI-Powered Search
Quantization is the transformation of numerical values into a smaller set of discrete values to reduce representation precision and resource requirements.
- How it is used
- Reduces embedding storage and similarity-search costs in dense vector retrieval.
- How deep it goes
- Explains scalar range mapping and benchmarks quantized indexes, including a combined optimization pipeline.
- Examples
- Uses outdoors post embeddings and queries such as “tent poles” to compare full-precision and Int8 indexes.
- What you can do after
- Apply it via FAISS and sentence-transformers to index embeddings and evaluate size, speed, and recall.
Deep Learning with Python, Third Edition
The technique of reducing numeric precision of model weights or activations to speed up computation while preserving output behavior as much as possible.
- How it is used
- Speeds up and shrinks inference models for deployment under memory, power, or latency constraints.
- How deep it goes
- Works through abs-max scaling, int8 matrix multiplication, output rescaling, and rounding error before recommending the Keras API.
- Examples
- Uses a small matrix multiplication to compare quantized and float32 outputs, then calls model.quantize("int8").
- What you can do after
- Apply it via Keras to an already trained model for inference and assess the accuracy tradeoff.
Read how Deep Learning with Python, Third Edition explains quantization →
Deep Learning with R, Third Edition
Model quantization is the conversion of model computations or parameters to lower-precision numerical representations to reduce resource requirements.
- How it is used
- Uses quantization to reduce deployment resource requirements and speed up inference.
- How deep it goes
- Works through scaling, rounding, int8 matrix multiplication, and unscaling before recommending the built-in API.
- Examples
- Uses a small matrix multiplication to compare quantized and float32 results, plus a Keras prediction workflow.
- What you can do after
- Apply it via Keras’s quantize_weights() to a trained model before prediction.
Read how Deep Learning with R, Third Edition explains quantization →
Math and Architectures of Deep Learning
The conversion of floating-point region boundaries or ratios into integer-aligned indices for feature extraction.
- How it is used
- Uses discretization to represent continuous values or align variable-sized RoIs with fixed pooling grids.
- How deep it goes
- Defines bucketization and integer rounding, explains their effects on probability distributions and RoI pooling, and defers RoIAlign details.
- Examples
- Uses height and weight buckets, fractional RoI dimensions, and fixed-size pooled feature maps as concrete examples.
- What you can do after
- Apply it via RoI pooling to turn variable-sized regions into fixed-sized feature representations.
Read how Math and Architectures of Deep Learning explains quantization →
Generative AI in Action
Quantization is the reduction of numerical precision used to represent model parameters, typically to reduce memory use and computational cost.
- How it is used
- Supports efficient LLM deployment and inference when memory, computation, energy, or GPU capacity constrain operations.
- How deep it goes
- Defines reduced-precision representation and explains benefits and risks, but stops short of a quantization workflow or hardware-specific implementation.
- Examples
- Uses Code Llama on a consumer GPU, QLoRA low-rank matrices, and DistilBERT compared with BERT.
- What you can do after
- Evaluate precision, memory, inference, energy, accuracy, and hardware-efficiency trade-offs when selecting quantized models.
Deep Learning with PyTorch, Second Edition
A model optimization technique that reduces numerical precision of parameters and operations to lower memory use and often speed up inference.
- How it is used
- Optimizes deployed models by reducing parameter and operation footprints for inference.
- How deep it goes
- Defines 8-bit conversion and demonstrates both PyTorch API usage and custom linear-layer implementation.
- Examples
- Uses serialized model-size measurement, custom int8 linear layers, and PyTorch dynamic quantization.
- What you can do after
- Apply it via PyTorch’s dynamic quantization API or a custom int8 implementation.
Read how Deep Learning with PyTorch, Second Edition explains quantization →
Transformers in Action
Quantization is the transformation of numerical model parameters or activations into lower-precision representations to reduce memory consumption or computational cost.
- How it is used
- Reduces large-language-model inference and fine-tuning resource demands through lower-precision weights and activations.
- How deep it goes
- Defines precision changes, quantifies memory savings, identifies range and accuracy risks, and demonstrates configuration without detailed quantization-error analysis.
- Examples
- Uses a data-type comparison table and a bitsandbytes configuration for loading Falcon 7B while measuring GPU memory.
- What you can do after
- Operate/configure bitsandbytes to load a 4-bit Falcon 7B model and measure its GPU memory impact.
The Complete Obsolete Guide to Generative AI
A quantized model is a machine-learning model whose numerical parameters or activations are represented at reduced precision to lower resource requirements.
- How it is used
- Enables local LLM deployment with less hardware by compressing models and preparing quantized weights.
- How deep it goes
- Demonstrates building, downloading, launching, and experimenting with quantized weights through command-line tooling.
- Examples
- Uses a RedPajama chat build, quantized weight download, launch command, and q4_1 output-type experiment.
- What you can do after
- Operate/configure it: build a compressed chat environment, obtain quantized weights, and test quantization-script arguments.
Read how The Complete Obsolete Guide to Generative AI explains quantization →
Time Series Forecasting Using Foundation Models
Quantization is the conversion of continuous numerical values into a finite set of discrete levels or tokens.
- How it is used
- Converts continuous time-series observations into language-model-compatible tokens for probabilistic forecasting.
- How deep it goes
- Explains uniform binning, clipping, precision loss, and decoding, while deferring the rationale for the clipping range.
- Examples
- Uses scaled time-series plots and contrasts 100-bin and 5-bin vocabularies to show retained shape and lost precision.
- What you can do after
- Apply uniform binning to represent scaled time-series values as tokens and recognize the resulting precision trade-off.
Read how Time Series Forecasting Using Foundation Models explains quantization →
Natural Language Processing in Action, Second Edition
Quantization is the transformation of numerical values into lower-precision discrete representations, often to reduce memory use and accelerate computation.
- How it is used
- Accelerates semantic-search indexing by shrinking vectors and replacing floating-point range work with faster integer comparisons.
- How deep it goes
- Defines scaling, rounding, integer storage, and product quantization, but stops short of detailed clustering or index implementation.
- Examples
- Demonstrates NumPy vectors, integer scaling, precision levels, and approximate-nearest-neighbor results on the New York Times dataset.
- What you can do after
- Apply it via NumPy-style scaling and integer conversion, and recognize product-quantization patterns in semantic-search indexes.
Read how Natural Language Processing in Action, Second Edition explains quantization →
Building Reliable AI Systems
Quantization reduces the numerical precision used to represent model values, often lowering storage or computation requirements.
- How it is used
- Reduces model-weight storage requirements in QLoRA fine-tuning and supports production-scale deployment.
- How deep it goes
- Explains 4-bit weights, NF4, double quantization, and higher-precision calculations, but does not detail implementation steps.
- Examples
- Uses an image-compression analogy and describes fine-tuning 7-billion- and 70-billion-parameter models on single GPUs.
- What you can do after
- Apply 4-bit quantization through QLoRA to reduce memory demands when fine-tuning large models.
Read how Building Reliable AI Systems explains quantization →
Related concepts
- QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that combines a quantized pretrained model with trainable low-rank adapters.
- post-training quantizationincludes · Post-training quantization converts a trained model’s parameters or computations to lower-precision representations without further model training.
- int4 quantizationincludes · INT4 quantization represents numerical values using 4-bit integers, reducing storage requirements while limiting numerical precision.
- block quantizationincludes · Block quantization partitions numerical values into groups and assigns quantization parameters, such as scale factors, separately to each group.
- per-channel quantizationincludes · Per-channel quantization assigns separate quantization parameters, such as scales, to individual channels along a tensor axis.
- quantization-aware trainingused by · Quantization-aware training optimizes a model while simulating or incorporating quantization effects so it can retain performance with reduced-precision representations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.













