If a model is too large or costly to deploy, why not make it smaller? Compression does that by shrinking, trimming, or approximating a model. The aim is faster deployment, lower latency, fewer expensive GPUs, and lower cost—without giving up too much of the behavior that made the large model useful. That balance remains an open problem. section 3.3 section 3.3.1
Three approaches illustrate different ways to save resources:
- Quantization stores numbers at lower precision to reduce memory requirements. The practical question is whether the discarded precision matters to the model’s results. section 3.3.1
- Post-training quantization can be applied to an existing model. A PyTorch example copies an FP32 model, prepares it, runs dataset inputs through it, and converts it. section 3.3.1
- Low-rank approximation represents a large matrix using smaller matrices. An SVD example reconstructs a 4 × 4 matrix at rank one, making the approximation visible rather than promising an exact copy. section 3.3.1
Listing 3.1 Example PTQ in PyTorch
import copy
import torch.ao.quantization as q
model_to_quantize = copy.deepcopy(model_fp32) #1
model_to_quantize.eval()
qconfig_mapping = q.get_default_qconfig_mapping("qnnpack") #2
prepared_model = q.prepare(model_to_quantize) #3
with torch.inference_mode(): #4
for x in dataset:
prepared_model(x)
model_quantized = q.convert(prepared_model) #5
The PyTorch listing gives a compact starting point for post-training quantization: prepare a model, pass dataset inputs through it, then convert it. section 3.3.1
Smaller representations keep pushing into experimental territory: beyond INT4, reported strategies reach INT2 and lower bit counts. Treat an impressive memory figure as a reason to test a model on the intended task, not as proof that its accuracy is acceptable. Also keep compression distinct from speculative decoding: the latter can speed sampling with a smaller assistant model but requires memory for both models. section 12.2.2 section 12.2.2
