Quantization
Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
LLMs in Production
Quantization is the conversion of model parameters or computations from higher-precision numerical representations to lower-precision representations.
- How it is used
- Makes LLM inference and deployment feasible when model size strains disk, RAM, VRAM, or serving budgets.
- How deep it goes
- Defines the precision trade-off and works through PyTorch and llama.cpp procedures, without deriving the underlying numerical mathematics.
- Examples
- Uses a dynamic INT8 PyTorch model and a 7B GGUF model on a Raspberry Pi to compare sizes and constraints.
- What you can do after
- Apply it via PyTorch or llama.cpp, then assess whether the smaller model fits hardware without unacceptable quality or speed losses.
Domain-Specific Small Language Models
Methods that reduce numeric precision or representational size of model values to improve efficiency, especially for inference; here the absmax method is a specific quantization procedure.
- How it is used
- Optimizes LLM inference for memory, latency, throughput, and hardware constraints.
- How deep it goes
- Explains precision formats, PTQ and QAT, schemes, hands-on absmax steps, 4-bit methods, and evaluation, without formal derivations.
- Examples
- Demonstrates FP32-to-INT8 GPT-2 conversion, GPTQ quantization of GPT-2 XL, and 8-bit antibody-model deployment.
- What you can do after
- Apply it via PyTorch, Transformers, AutoGPTQ, BitsAndBytes, or ONNX-compatible runtimes, then evaluate quality and serving performance.
Read how Domain-Specific Small Language Models explains quantization →
AI-Powered Search
Quantization is the transformation of numerical values into a smaller set of discrete values to reduce representation precision and resource requirements.
- How it is used
- Reduces embedding storage and search costs in dense-vector retrieval while trading against recall.
- How deep it goes
- Works through scalar precision mapping and an Int8 indexing benchmark, then shows combined optimizations without detailing product quantization.
- Examples
- Uses outdoor-post embeddings, sample search queries, FAISS indexes, and measured index size and recall.
- What you can do after
- Apply it via Sentence Transformers and FAISS to build an Int8 index and compare its size and recall with full precision.
Deep Learning with Python, Third Edition
The technique of reducing numeric precision of model weights or activations to speed up computation while preserving output behavior as much as possible.
- How it is used
- Reduces the size of an inference model and can speed predictions for constrained devices or low-latency applications.
- How deep it goes
- Works through scaling, int8 matrix multiplication, rescaling, and rounding error, then recommends a built-in API over manual implementation.
- Examples
- Uses a two-tensor matrix multiplication, an abs-max scaling function, output comparisons, and a Keras prediction call.
- What you can do after
- Apply it via Keras model.quantize("int8") to prepare a trained model for inference.
Read how Deep Learning with Python, Third Edition explains quantization →
Deep Learning with R, Third Edition
Model quantization is the conversion of model computations or parameters to lower-precision numerical representations to reduce resource requirements.
- How it is used
- Uses int8 quantization to reduce inference-model size and speed inference when deployment resources or latency matter.
- How deep it goes
- Works through scaling, int8 matrix multiplication, unscaling, and numerical comparison, then favors a built-in API over hand implementation.
- Examples
- Compares quantized and float32 results for small matrices and shows R code for scaling, multiplication, and Keras predictions.
- What you can do after
- Apply it via Keras quantize_weights("int8") to prepare an inference-only model and run predictions.
Read how Deep Learning with R, Third Edition explains quantization →
Math and Architectures of Deep Learning
The conversion of floating-point region boundaries or ratios into integer-aligned indices for feature extraction.
- How it is used
- Uses discretization to represent continuous values or align variable-sized RoIs with fixed pooling grids.
- How deep it goes
- Defines bucketization and integer rounding, explains their effects on probability distributions and RoI pooling, and defers RoIAlign details.
- Examples
- Uses height and weight buckets, fractional RoI dimensions, and fixed-size pooled feature maps as concrete examples.
- What you can do after
- Apply it via RoI pooling to turn variable-sized regions into fixed-sized feature representations.
Read how Math and Architectures of Deep Learning explains quantization →
Generative AI in Action
Quantization is the reduction of numerical precision used to represent model parameters, typically to reduce memory use and computational cost.
- How it is used
- Supports efficient LLM deployment and inference when memory, computation, energy, or GPU capacity constrain operations.
- How deep it goes
- Defines reduced-precision representation and explains benefits and risks, but stops short of a quantization workflow or hardware-specific implementation.
- Examples
- Uses Code Llama on a consumer GPU, QLoRA low-rank matrices, and DistilBERT compared with BERT.
- What you can do after
- Evaluate precision, memory, inference, energy, accuracy, and hardware-efficiency trade-offs when selecting quantized models.
Deep Learning with PyTorch, Second Edition
A model optimization technique that reduces numerical precision of parameters and operations to lower memory use and often speed up inference.
- How it is used
- Optimizes deployed models by reducing parameter and operation footprints for inference.
- How deep it goes
- Defines 8-bit conversion and demonstrates both PyTorch API usage and custom linear-layer implementation.
- Examples
- Uses serialized model-size measurement, custom int8 linear layers, and PyTorch dynamic quantization.
- What you can do after
- Apply it via PyTorch’s dynamic quantization API or a custom int8 implementation.
Read how Deep Learning with PyTorch, Second Edition explains quantization →
Transformers in Action
Quantization is the transformation of numerical model parameters or activations into lower-precision representations to reduce memory consumption or computational cost.
- How it is used
- Reduces large-language-model inference and fine-tuning resource demands through lower-precision weights and activations.
- How deep it goes
- Defines precision changes, quantifies memory savings, identifies range and accuracy risks, and demonstrates configuration without detailed quantization-error analysis.
- Examples
- Uses a data-type comparison table and a bitsandbytes configuration for loading Falcon 7B while measuring GPU memory.
- What you can do after
- Operate/configure bitsandbytes to load a 4-bit Falcon 7B model and measure its GPU memory impact.
The Complete Obsolete Guide to Generative AI
A quantized model is a machine-learning model whose numerical parameters or activations are represented at reduced precision to lower resource requirements.
- How it is used
- Enables local LLM deployment with less hardware by compressing models and preparing quantized weights.
- How deep it goes
- Demonstrates building, downloading, launching, and experimenting with quantized weights through command-line tooling.
- Examples
- Uses a RedPajama chat build, quantized weight download, launch command, and q4_1 output-type experiment.
- What you can do after
- Operate/configure it: build a compressed chat environment, obtain quantized weights, and test quantization-script arguments.
Read how The Complete Obsolete Guide to Generative AI explains quantization →
Time Series Forecasting Using Foundation Models
Quantization is the conversion of continuous numerical values into a finite set of discrete levels or tokens.
- How it is used
- Converts continuous time-series observations into language-model-compatible tokens for probabilistic forecasting.
- How deep it goes
- Explains uniform binning, clipping, precision loss, and decoding, while deferring the rationale for the clipping range.
- Examples
- Uses scaled time-series plots and contrasts 100-bin and 5-bin vocabularies to show retained shape and lost precision.
- What you can do after
- Apply uniform binning to represent scaled time-series values as tokens and recognize the resulting precision trade-off.
Read how Time Series Forecasting Using Foundation Models explains quantization →
Natural Language Processing in Action, Second Edition
Quantization is the transformation of numerical values into lower-precision discrete representations, often to reduce memory use and accelerate computation.
- How it is used
- Accelerates semantic-search indexing by shrinking vectors and replacing floating-point range work with faster integer comparisons.
- How deep it goes
- Defines scaling, rounding, integer storage, and product quantization, but stops short of detailed clustering or index implementation.
- Examples
- Demonstrates NumPy vectors, integer scaling, precision levels, and approximate-nearest-neighbor results on the New York Times dataset.
- What you can do after
- Apply it via NumPy-style scaling and integer conversion, and recognize product-quantization patterns in semantic-search indexes.
Read how Natural Language Processing in Action, Second Edition explains quantization →
Building Reliable AI Systems
Quantization reduces the numerical precision used to represent model values, often lowering storage or computation requirements.
- How it is used
- Reduces model-weight storage requirements in QLoRA fine-tuning and supports production-scale deployment.
- How deep it goes
- Explains 4-bit weights, NF4, double quantization, and higher-precision calculations, but does not detail implementation steps.
- Examples
- Uses an image-compression analogy and describes fine-tuning 7-billion- and 70-billion-parameter models on single GPUs.
- What you can do after
- Apply 4-bit quantization through QLoRA to reduce memory demands when fine-tuning large models.
Read how Building Reliable AI Systems explains quantization →
Related concepts
- QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
- scalar quantizationincludes · Scalar quantization maps numerical scalar values to a finite set of discrete levels, typically reducing their precision or range.
- QA-LoRAused by · QA-LoRA is a low-rank adaptation method that integrates quantization-aware operations and group-wise processing to support efficient low-bit adaptation.
- post-training quantizationincludes · Post-training quantization is the conversion of a trained model’s parameters or computations to lower-precision representations without additional training.
- quantization-aware trainingincludes · Quantization-aware training optimizes a model while simulating or incorporating reduced-precision effects so it adapts to quantized representations.
- PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.