Scalar quantization compresses each embedding component independently, mapping higher-precision values to lower-precision representations. It reduces vector memory use and similarity-computation cost while trading away some recall. section 13.7 section 13.7.1
The mechanism proceeds in progressively smaller numeric representations:
- Map Float32 values to Float16 for approximately similar values at half the storage cost. section 13.7.1
- Map floating-point ranges to Int8 values from -128 to 127, or to 0 through 255 for unsigned integers. section 13.7.1
- Clamp extremes and use the original value density to occupy the quantized range more effectively. section 13.7.1
Int8 vector-search trade-off
For 18,456 1,024-dimensional embeddings, an Int8 FAISS index measured 18.91 MB instead of 75.6 MB. Search improved by 38.65%, recall was 0.9289, and full-precision reranking restored recall to 1.0.
A practical FAISS workflow quantizes embeddings with sentence_transformers, adds them to an inner-product index, persists the index, quantizes query vectors, and compares optimized results with the full-precision baseline. Reranking shortlisted results with Float32 embeddings can recover relevance.
Figure 13.9 Quantizing data: full precision, reduced scalar precision, and binary precision
Compare full precision, reduced scalar precision, and binary precision visually before choosing a storage and recall trade-off. section 13.7
