Large vector indexes can consume tens or hundreds of gigabytes, creating storage, latency, and cost bottlenecks. Vector compression reduces embedding memory while aiming to retain the information retrieval needs. section 4.6.2
Common approaches represent vectors more compactly:
- Product quantization approximates vectors using smaller codebook-based representations. section 4.6.2
- Scalar quantization stores floats in smaller data types such as int8; binary quantization reduces each dimension to one bit. section 4.6.2
Compressed first pass, precise reranking
Search compressed vectors to find candidate documents, then pass those candidates to a cross-encoder reranker. This multistage approach supports scale while recovering precision in the second stage. section 4.6.2
Compression is most useful for growing corpora or memory-constrained hardware, but may be unnecessary for small applications. FAISS can quantize existing vectors; Qdrant and Weaviate expose compression configuration options. section 4.6.2
