Scalar quantization
Scalar quantization maps numerical scalar values to a finite set of discrete levels, typically reducing their precision or range.
AI-Powered Search
Scalar quantization maps individual numerical vector components from a higher-precision representation to a lower-precision scalar format.
- How it is used
- Optimizes large-scale vector search by reducing embedding storage and similarity-computation costs.
- How deep it goes
- Explains independent value mapping, range clamping, calibration, indexing, benchmarking, and reranking, stopping short of quantization algorithm derivations.
- Examples
- Uses Float32-to-Float16 values, Float32-to-Int8 mapping, FAISS indexes, outdoor-search queries, and benchmark measurements.
- What you can do after
- Apply it via FAISS and sentence_transformers, then measure index size, search speed, recall, and reranked recall.
Build a DeepSeek Model (From Scratch)
Standard quantization maps an entire tensor to a lower-precision numerical range using one shared scaling factor.
- How it is used
- Converts high-precision model tensors into lower-precision numerical formats for efficient representation.
- How deep it goes
- Explains maximum-based scaling, rounding, dequantization, and outlier-induced error through a worked vector example.
- Examples
- Uses the vector [2.0, 3.0, 500.0] and a signed 8-bit range to quantify precision loss.
- What you can do after
- Apply shared scaling and dequantization while diagnosing precision loss caused by tensor outliers.
Read how Build a DeepSeek Model (From Scratch) explains scalar quantization →
What to know first
- rerankingReranking is the process of rescoring or reordering an initially retrieved or ranked set of candidates using an additional assessment of relevance or priority.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
Related concepts
- quantizationkind of · Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- vector compressionkind of · Vector compression reduces the storage required for numerical vector representations, typically by encoding them with fewer or lower-precision values while retaining useful information.
- fine-grained quantizationcontrasts with · Fine-grained quantization reduces numerical precision independently within small groups of activations or blocks of weights, assigning each region its own scaling factor.
Questions about scalar quantization
What is scalar quantization?
Scalar quantization maps numerical scalar values to a finite set of discrete levels, typically reducing their precision or range.
Which Manning books explain scalar quantization?
AI-Powered Search and Build a DeepSeek Model (From Scratch).
What should I know before learning scalar quantization?
Manning books that teach scalar quantization usually introduce reranking and embedding first.
What is the difference between scalar quantization and fine-grained quantization?
Scalar quantization maps numerical scalar values to a finite set of discrete levels, typically reducing their precision or range. Fine-grained quantization reduces numerical precision independently within small groups of activations or blocks of weights, assigning each region its own scaling factor.
What is scalar quantization a kind of?
Scalar quantization is a kind of quantization. Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

