liveBook · Manning

State_dict

As explained in Quantization and Fast Inference

state_dict is a PyTorch interface for representing a module's parameters and persistent buffers as a mapping of names to tensors.

A state_dict is the portable serialization path for quantized PyTorch models: save the mapping, reconstruct the quantized model skeleton, then load the mapping. This avoids relying on full-model pickling, which can fail across PyTorch versions. section 6.2

Listing 6.3 Save/load round-trip for TorchAO models
# Save
torch.save(model.state_dict(), "bert_int8wo.pt")                  #A
 
# Load
loaded = AutoModelForSequenceClassification.from_pretrained(...)
quantize_(loaded, Int8WeightOnlyConfig())                          #B
loaded.load_state_dict(
    torch.load("bert_int8wo.pt", weights_only=False)               #C
)

Follow the save/load round-trip, including reconstruction of the quantized model before loading its state. section 6.2

How Quantization and Fast Inference covers it

How deep it goes
Shows save/load mechanics and quantized-skeleton reconstruction without detailing the mapping's internal format.
Examples
Demonstrates a BERT-base SST-2 round-trip that retains 92.32% accuracy before and after saving.
What it assumes
Reconstruct the quantized model skeleton before loading the saved state.
What you can do after
Operate PyTorch save/load to serialize and restore a quantized model after rebuilding its quantized skeleton.
How it connects
Connects PyTorch serialization with TorchAO quantization and AffineQuantizedTensor.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.