Weight-only quantization stores model weights at reduced precision while keeping activations at higher precision. Its primary benefit is lower weight memory and bandwidth, not guaranteed faster arithmetic. In bandwidth-bound decoding, reading fewer weight bytes can improve throughput; on other workloads, dequantization and runtime overhead can outweigh the savings. section 3.1 section 11.3.3 section 6.2
Weights are fixed after training, so a weight-only configuration can quantize them without activation calibration data. In the TorchAO path, quantize_() replaces each nn.Linear weight with an INT8 AffineQuantizedTensor containing per-channel scales; the module itself remains nn.Linear. Unlike a path that quantizes both operands for integer GEMM, weight-only execution may dequantize weights before multiplication. section 6.2 section 3.1
Listing 6.2 TorchAO quantization in one line
from torchao.quantization import quantize_, Int8WeightOnlyConfig
model = AutoModelForSequenceClassification.from_pretrained(
"textattack/bert-base-uncased-SST-2"
)
quantize_(model, Int8WeightOnlyConfig()) #AApply TorchAO INT8 weight-only quantization to a loaded BERT model. section 6.2
Predict compression from the fraction of parameters the configuration can reach. If fraction p is converted from FP32 to INT8 and the rest remains FP32, the ideal compression is 1 / (0.25p + 1 − p), before scale overhead. For BERT, p = 0.782 gives 2.42×; the measured result is 2.40×. For TinyLlama, per-channel scale storage contributes to a larger gap: 3.38× measured versus 3.57× ideal. section 6.2
Table 6.1 Architecture determines compression ceiling
|
Model
|
Linear layers
|
Linear params (%)
|
Theoretical INT8 ceiling
|
Measured
|
|
ResNet-18
|
1
|
513,000 (4.4%)
|
1.03×
|
1.03×
|
|
BERT-base
|
74
|
85,609,730 (78.2%)
|
2.42×
|
2.40×
|
|
TinyLlama-1.1B
|
155
|
1,034,420,224 (94.0%)
|
3.57×
|
3.38×
|
Compare the predicted and measured compression across linear- and convolution-dominated architectures. section 6.2
Measure quality and latency separately
On a T4, BERT-base INT8 weight-only shrank from 417.73 MB to 174.05 MB and SST-2 accuracy changed from 92.43% to 92.32%. Eager latency nevertheless rose from 9.33 ms to 17.69 ms. Smaller weights alone did not make this runtime faster. section 6.2
Bandwidth-bound decoding
On an M3, Llama-3.2-1B FP16 weights occupy roughly 2.5 GB; four-bit weights reduce the approximate weight read per token to 625 MB. MLX q4 achieved 78.5 tokens/s at a 32-token prompt versus 29.5 for FP16. At 1,024 tokens, q4 fell to 50.4 tokens/s as KV-cache and activation traffic became more significant. section 11.3.3
Freed weight memory can also enlarge a serving system’s KV-cache pool, but extra capacity matters only when the workload approaches its limit. In a Qwen2.5-3B-Instruct sweep on a 24 GB L4, both FP16 and AWQ four-bit kept queue depth at zero and KV occupancy near zero; the expected saturation shift was not observed. section 12.3 section 12.3.2
Choose the configuration after counting eligible parameters, then measure quality, latency, throughput, and memory on the target runtime. TorchAO’s cited weight-only path reaches nn.Linear layers, not Conv2d weights, so a convolution-dominated model needs a different quantization path for meaningful compression. section 6.2
