liveBook · Manning

Quantize_()

As explained in Quantization and Fast Inference

quantize_ is a TorchAO API that replaces eligible model weights with quantized tensor representations.

TorchAO’s quantize_() walks a model’s module tree and replaces eligible nn.Linear weights with AffineQuantizedTensor representations storing INT8 data and per-channel scales. The modules remain nn.Linear; weight-only configurations need no calibration data. section 6.2

Compression depends on how many parameters belong to nn.Linear. Conv2d weights remain untouched: the stable configurations reshape weights to two dimensions, which does not fit Conv2d’s four-dimensional weights. section 6.2

Table 6.1 Architecture determines compression ceiling
Model
Linear layers
Linear params (%)
Theoretical INT8 ceiling
Measured
ResNet-18
1
513,000 (4.4%)
1.03×
1.03×
BERT-base
74
85,609,730 (78.2%)
2.42×
2.40×
TinyLlama-1.1B
155
1,034,420,224 (94.0%)
3.57×
3.38×

Compare parameter distribution with theoretical and measured INT8 compression across ResNet-18, BERT-base, and TinyLlama. section 6.2

How Quantization and Fast Inference covers it

How deep it goes
Traces module replacement and compression limits; leaves Conv2d quantization to another API.
Examples
Compares ResNet-18, BERT-base, and TinyLlama, reporting compression from 1.03× to 3.38×.
What it assumes
Assumes a PyTorch module tree; weight-only configurations require no calibration data.
What you can do after
Apply TorchAO’s quantize_() to eligible nn.Linear weights and estimate compression from parameter distribution.
How it connects
Connects INT8 weight-only quantization with nn.Linear, AffineQuantizedTensor, and PT2E export quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.