TurboQuant reduces KV-cache memory for long-context serving, where the cache can rival model size. It quantizes vectors as they arrive, without offline calibration, and can be combined with a weight-quantization method. section 7.0 section 7.5
For each key or value vector, store its L2 norm in FP16 and normalize the vector. Apply a random orthogonal rotation to redistribute its direction across coordinates, then quantize each rotated coordinate using a fixed Lloyd–Max codebook. Because the rotation makes the coordinate distribution predictable across input vectors, the codebook needs no per-vector calibration. section 7.5
The deployment tradeoff is cache capacity versus reconstruction quality:
- At head dimension 128, TQ4 uses 132 bytes per token for keys and values, about 3.88× compression versus FP16. section 7.5
- TQ3 uses 100 bytes per token, about 5.12× compression. On OPT-6.7B WikiText-2 evaluation, perplexity rises from 16.31 in FP16 to 16.43 for TQ4 and 17.05 for TQ3. section 7.5
- Reconstruction error is asymmetric: TQ4 key-cache MSE is 0.0756, versus 0.0056 for values; TQ3 roughly triples both. section 7.5
Choosing a bit width
For a 128-dimensional head, choosing TQ3 instead of TQ4 saves 32 bytes per token across keys and values, while the cited evaluation shows a larger perplexity increase: +4.5% versus FP16 for TQ3, compared with +0.7% for TQ4. section 7.5
Listing 7.13 TurboQuant vector quantization (Algorithm 1)
class TurboQuantizer:
def __init__(self, head_dim, codebooks, seed=42, device="cuda"):
self.head_dim = head_dim
gen = torch.Generator()
gen.manual_seed(seed)
G = torch.randn(head_dim, head_dim, generator=gen)
Pi, _ = torch.linalg.qr(G) #A
self.Pi = Pi.to(device)
self.centroids = {}
self.boundaries = {}
for bits, cb in codebooks.items():
self.centroids[bits] = torch.tensor(
cb["centroids"], dtype=torch.float32, device=device)
self.boundaries[bits] = torch.tensor(
cb["boundaries"], dtype=torch.float32, device=device)
def quantize(self, vectors, bits):
shape = vectors.shape
flat = vectors.reshape(-1, self.head_dim).float()
norms = torch.norm(flat, dim=1, keepdim=True) #B
normalized = flat / torch.clamp(norms, min=1e-8) #C
rotated = normalized @ self.Pi.T #D
interior_b = self.boundaries[bits][1:-1]
indices = torch.searchsorted(interior_b, rotated) #E
return {"indices": indices, "norms": norms.half(),
"bits": bits, "shape": shape}
def dequantize(self, compressed):
c = self.centroids[compressed["bits"]]
recon_rot = c[compressed["indices"]] #F
recon_norm = recon_rot @ self.Pi #G
recon = recon_norm * compressed["norms"].float() #H
return recon.reshape(compressed["shape"])
Inspect the PyTorch quantizer implementation, which builds a seeded random orthogonal matrix and stores codebook centroids and boundaries. section 7.5
