liveBook · Manning

Weight-only quantization

As explained in Quantization and Fast Inference

Reducing the precision of model weights while leaving activations at higher precision.

Weight-only quantization stores model weights at reduced precision while keeping activations at higher precision. Its primary benefit is lower weight memory and bandwidth, not guaranteed faster arithmetic. In bandwidth-bound decoding, reading fewer weight bytes can improve throughput; on other workloads, dequantization and runtime overhead can outweigh the savings. section 3.1 section 11.3.3 section 6.2

Weights are fixed after training, so a weight-only configuration can quantize them without activation calibration data. In the TorchAO path, quantize_() replaces each nn.Linear weight with an INT8 AffineQuantizedTensor containing per-channel scales; the module itself remains nn.Linear. Unlike a path that quantizes both operands for integer GEMM, weight-only execution may dequantize weights before multiplication. section 6.2 section 3.1

Listing 6.2 TorchAO quantization in one line
from torchao.quantization import quantize_, Int8WeightOnlyConfig
 
model = AutoModelForSequenceClassification.from_pretrained(
    "textattack/bert-base-uncased-SST-2"
)
quantize_(model, Int8WeightOnlyConfig())                           #A

Apply TorchAO INT8 weight-only quantization to a loaded BERT model. section 6.2

Predict compression from the fraction of parameters the configuration can reach. If fraction p is converted from FP32 to INT8 and the rest remains FP32, the ideal compression is 1 / (0.25p + 1 − p), before scale overhead. For BERT, p = 0.782 gives 2.42×; the measured result is 2.40×. For TinyLlama, per-channel scale storage contributes to a larger gap: 3.38× measured versus 3.57× ideal. section 6.2

Table 6.1 Architecture determines compression ceiling
Model
Linear layers
Linear params (%)
Theoretical INT8 ceiling
Measured
ResNet-18
1
513,000 (4.4%)
1.03×
1.03×
BERT-base
74
85,609,730 (78.2%)
2.42×
2.40×
TinyLlama-1.1B
155
1,034,420,224 (94.0%)
3.57×
3.38×

Compare the predicted and measured compression across linear- and convolution-dominated architectures. section 6.2

Measure quality and latency separately

On a T4, BERT-base INT8 weight-only shrank from 417.73 MB to 174.05 MB and SST-2 accuracy changed from 92.43% to 92.32%. Eager latency nevertheless rose from 9.33 ms to 17.69 ms. Smaller weights alone did not make this runtime faster. section 6.2

Bandwidth-bound decoding

On an M3, Llama-3.2-1B FP16 weights occupy roughly 2.5 GB; four-bit weights reduce the approximate weight read per token to 625 MB. MLX q4 achieved 78.5 tokens/s at a 32-token prompt versus 29.5 for FP16. At 1,024 tokens, q4 fell to 50.4 tokens/s as KV-cache and activation traffic became more significant. section 11.3.3

Freed weight memory can also enlarge a serving system’s KV-cache pool, but extra capacity matters only when the workload approaches its limit. In a Qwen2.5-3B-Instruct sweep on a 24 GB L4, both FP16 and AWQ four-bit kept queue depth at zero and KV occupancy near zero; the expected saturation shift was not observed. section 12.3 section 12.3.2

Choose the configuration after counting eligible parameters, then measure quality, latency, throughput, and memory on the target runtime. TorchAO’s cited weight-only path reaches nn.Linear layers, not Conv2d weights, so a convolution-dominated model needs a different quantization path for meaningful compression. section 6.2

How Quantization and Fast Inference covers it

How deep it goes
Explains per-channel weight storage and compression ceilings, then measures accuracy, latency, throughput, and serving headroom.
Examples
Compares TorchAO BERT on a T4, MLX Llama decoding on an M3, and FP16 versus AWQ serving on an L4.
What you can do after
Apply it via TorchAO and evaluate compression, quality, and latency on the target runtime.
How it connects
Connects activation quantization, calibration, integer GEMM, per-channel quantization, and KV-cache capacity.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.