liveBook · Manning

AffineQuantizedTensor

As explained in Quantization and Fast Inference

AffineQuantizedTensor is a TorchAO tensor subclass that represents quantized values together with their quantization parameters.

AffineQuantizedTensor stores INT8 weight data with per-channel FP32 scales. TorchAO’s quantize_() replaces nn.Linear weights with this subclass while leaving the module structure unchanged; the scales add storage overhead. section 6.2

BERT weight representation

A BERT query weight is represented with shape [768, 768], block_size=(1, 768), and int_dtype=torch.int8.

How Quantization and Fast Inference covers it

How deep it goes
Defines the tensor’s storage and module replacement, and notes scale overhead without detailing quantization arithmetic.
Examples
Shows BERT query and key weights with shape [768, 768], block_size=(1, 768), and int8 storage.
What you can do after
Evaluate expected storage gains against per-channel scale overhead when quantizing nn.Linear weights with TorchAO.
How it connects
Connects TorchAO quantize_(), PyTorch nn.Linear, INT8 weight-only quantization, and per-channel scales.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.