liveBook · Manning

Scalar quantization

Scalar quantization maps numerical scalar values to a finite set of discrete levels, typically reducing their precision or range.

How 2 Manning books explain it

AI-Powered Search

Scalar quantization maps individual numerical vector components from a higher-precision representation to a lower-precision scalar format.

How it is used
Optimizes large-scale vector search by reducing embedding storage and similarity-computation costs.
How deep it goes
Explains independent value mapping, range clamping, calibration, indexing, benchmarking, and reranking, stopping short of quantization algorithm derivations.
Examples
Uses Float32-to-Float16 values, Float32-to-Int8 mapping, FAISS indexes, outdoor-search queries, and benchmark measurements.
What you can do after
Apply it via FAISS and sentence_transformers, then measure index size, search speed, recall, and reranked recall.

Read how AI-Powered Search explains scalar quantization →

Build a DeepSeek Model (From Scratch)

Standard quantization maps an entire tensor to a lower-precision numerical range using one shared scaling factor.

How it is used
Converts high-precision model tensors into lower-precision numerical formats for efficient representation.
How deep it goes
Explains maximum-based scaling, rounding, dequantization, and outlier-induced error through a worked vector example.
Examples
Uses the vector [2.0, 3.0, 500.0] and a signed 8-bit range to quantify precision loss.
What you can do after
Apply shared scaling and dequantization while diagnosing precision loss caused by tensor outliers.

Read how Build a DeepSeek Model (From Scratch) explains scalar quantization →

What to know first

Related concepts

Questions about scalar quantization

What is scalar quantization?

Scalar quantization maps numerical scalar values to a finite set of discrete levels, typically reducing their precision or range.

Which Manning books explain scalar quantization?

AI-Powered Search and Build a DeepSeek Model (From Scratch).

What should I know before learning scalar quantization?

Manning books that teach scalar quantization usually introduce reranking and embedding first.

What is the difference between scalar quantization and fine-grained quantization?

Scalar quantization maps numerical scalar values to a finite set of discrete levels, typically reducing their precision or range. Fine-grained quantization reduces numerical precision independently within small groups of activations or blocks of weights, assigning each region its own scaling factor.

What is scalar quantization a kind of?

Scalar quantization is a kind of quantization. Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.