liveBook · Manning

TurboQuant

As explained in Quantization and Fast Inference

TurboQuant is an online quantization method that compresses key-value caches using random rotations and a fixed codebook without calibration.

TurboQuant reduces KV-cache memory for long-context serving, where the cache can rival model size. It quantizes vectors as they arrive, without offline calibration, and can be combined with a weight-quantization method. section 7.0 section 7.5

For each key or value vector, store its L2 norm in FP16 and normalize the vector. Apply a random orthogonal rotation to redistribute its direction across coordinates, then quantize each rotated coordinate using a fixed Lloyd–Max codebook. Because the rotation makes the coordinate distribution predictable across input vectors, the codebook needs no per-vector calibration. section 7.5

The deployment tradeoff is cache capacity versus reconstruction quality:

  • At head dimension 128, TQ4 uses 132 bytes per token for keys and values, about 3.88× compression versus FP16. section 7.5
  • TQ3 uses 100 bytes per token, about 5.12× compression. On OPT-6.7B WikiText-2 evaluation, perplexity rises from 16.31 in FP16 to 16.43 for TQ4 and 17.05 for TQ3. section 7.5
  • Reconstruction error is asymmetric: TQ4 key-cache MSE is 0.0756, versus 0.0056 for values; TQ3 roughly triples both. section 7.5

Choosing a bit width

For a 128-dimensional head, choosing TQ3 instead of TQ4 saves 32 bytes per token across keys and values, while the cited evaluation shows a larger perplexity increase: +4.5% versus FP16 for TQ3, compared with +0.7% for TQ4. section 7.5

Listing 7.13 TurboQuant vector quantization (Algorithm 1)
class TurboQuantizer:
    def __init__(self, head_dim, codebooks, seed=42, device="cuda"):
        self.head_dim = head_dim
        gen = torch.Generator()
        gen.manual_seed(seed)
        G = torch.randn(head_dim, head_dim, generator=gen)
        Pi, _ = torch.linalg.qr(G)                          #A
        self.Pi = Pi.to(device)
        self.centroids = {}
        self.boundaries = {}
        for bits, cb in codebooks.items():
            self.centroids[bits] = torch.tensor(
                cb["centroids"], dtype=torch.float32, device=device)
            self.boundaries[bits] = torch.tensor(
                cb["boundaries"], dtype=torch.float32, device=device)
 
    def quantize(self, vectors, bits):
        shape = vectors.shape
        flat = vectors.reshape(-1, self.head_dim).float()
        norms = torch.norm(flat, dim=1, keepdim=True)        #B
        normalized = flat / torch.clamp(norms, min=1e-8)     #C
        rotated = normalized @ self.Pi.T                     #D
        interior_b = self.boundaries[bits][1:-1]
        indices = torch.searchsorted(interior_b, rotated)    #E
        return {"indices": indices, "norms": norms.half(),
                "bits": bits, "shape": shape}
 
    def dequantize(self, compressed):
        c = self.centroids[compressed["bits"]]
        recon_rot = c[compressed["indices"]]                 #F
        recon_norm = recon_rot @ self.Pi                     #G
        recon = recon_norm * compressed["norms"].float()     #H
        return recon.reshape(compressed["shape"])
 

Inspect the PyTorch quantizer implementation, which builds a seeded random orthogonal matrix and stores codebook centroids and boundaries. section 7.5

How Quantization and Fast Inference covers it

How deep it goes
Explains norm separation, random rotation, codebook quantization, and memory-quality tradeoffs, with a PyTorch implementation.
Examples
Compares TQ3 and TQ4 memory and WikiText-2 perplexity on OPT-6.7B, and reports key-versus-value reconstruction error.
What you can do after
Apply online vector quantization to key and value vectors using a random rotation and fixed codebook.
How it connects
Connects vector quantization and random rotation to KV-cache compression and composes with weight-quantization methods.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.