liveBook · Manning

Quantized Johnson-Lindenstrauss correction

As explained in Quantization and Fast Inference

A quantized Johnson-Lindenstrauss correction uses quantized random projections to produce unbiased estimates of inner products.

A Quantized Johnson–Lindenstrauss (QJL) correction reserves one bit to produce unbiased inner-product estimates, useful when ranking fidelity matters in vector search. For KV-cache attention, those estimates pass through softmax: its exponential amplifies variance, and QJL consistently worsens perplexity. Use TurboQuant’s MSE-only algorithm for KV caches. section 7.5

Figure 7.19 Per-layer reconstruction MSE on OPT-6.7B with and without QJL correction. MSE-only (solid lines) beats QJL-corrected (dashed lines) across all 32 layers, both bit-widths, and both keys and values. QJL TQ4 is worse than MSE-only TQ3 in most key cache layers. Log-scale Y-axis.

Compare per-layer reconstruction error with and without QJL correction across keys, values, and bit-widths. section 7.5

How Quantization and Fast Inference covers it

How deep it goes
Explains the one-bit correction and why softmax variance amplification makes it degrade KV-cache perplexity.
Examples
Compares QJL-corrected and MSE-only reconstruction across 32 OPT-6.7B layers, including both keys and values.
What you can do after
Apply MSE-only TurboQuant to KV caches and reserve QJL for settings where unbiased inner-product estimates support ranking.
How it connects
Connects quantized inner products to TurboQuant, vector search, KV caches, and softmax attention.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.