liveBook · Manning

Compression ratio

As explained in Quantization and Fast Inference

Compression ratio is the factor by which a model's stored representation is reduced relative to an original representation.

Compression ratio measures how many times smaller a model’s stored representation is than its original. For INT8 quantization of a fraction p of FP32 parameters, the theoretical ceiling is 1 / (0.25p + 1 − p); unquantized parameters remain full size. section 6.2

Architecture changes the result

TorchAO INT8 weight-only quantization measured 1.03× compression for ResNet-18, 2.40× for BERT-base, and 3.38× for TinyLlama-1.1B. The results track how many parameters are in nn.Linear layers; per-channel scales also add storage overhead. section 6.2

Table 6.1 Architecture determines compression ceiling
Model
Linear layers
Linear params (%)
Theoretical INT8 ceiling
Measured
ResNet-18
1
513,000 (4.4%)
1.03×
1.03×
BERT-base
74
85,609,730 (78.2%)
2.42×
2.40×
TinyLlama-1.1B
155
1,034,420,224 (94.0%)
3.57×
3.38×

Compare each model’s nn.Linear parameter share with its theoretical INT8 ceiling and measured compression. section 6.2

How Quantization and Fast Inference covers it

How deep it goes
Calculates a theoretical INT8 ceiling from parameter shares and compares it with measured results, including scale overhead.
Examples
Compares measured compression for ResNet-18, BERT-base, and TinyLlama-1.1B.
What you can do after
Evaluate likely TorchAO INT8 weight-only size reduction by calculating the share of parameters in nn.Linear layers.
How it connects
Connects quantization, INT8 storage, FP32 parameters, and model architecture.

Also explained in

Compare how 2 books explain compression ratio →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.