Post-training quantization
Post-training quantization is the conversion of a trained model’s parameters or computations to lower-precision representations without additional training.
Domain-Specific Small Language Models
A quantization method applied after training to reduce model numeric precision, typically to improve deployment efficiency.
- How it is used
- Reduces language-model memory footprint and improves deployment efficiency across constrained inference hardware.
- How deep it goes
- Explains zero-point and absmax formulas, then demonstrates INT8 conversion and evaluation, while deferring advanced techniques to later treatment.
- Examples
- Uses FP32-to-INT8 GPT-2 conversion, absmax scaling, weight histograms, generated continuations, and perplexity scores.
- What you can do after
- Apply it with PyTorch and Transformers, then evaluate quantization effects using generated text, weight distributions, and perplexity.
Read how Domain-Specific Small Language Models explains post-training quantization →
Transformers in Action
Posttraining quantization converts a pretrained model to lower-precision representations without additional model training.
- How it is used
- Reduces model inference memory when computational resources are limited or deployment must happen quickly.
- How deep it goes
- Compares quantization variants and trade-offs, but does not detail the quantization algorithm’s internal calculations.
- Examples
- Uses a Falcon 7B bitsandbytes configuration and a data-type comparison spanning float32 through int4.
- What you can do after
- Apply it via bitsandbytes by configuring 4-bit loading and measuring GPU memory usage.
Read how Transformers in Action explains post-training quantization →
Related concepts
- GPTQincludes · GPTQ is a one-shot post-training weight-quantization method that uses calibration data and approximate second-order information to reduce precision while limiting accuracy loss.
- quantizationkind of · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
- quantization-aware trainingalternative to · Quantization-aware training optimizes a model while simulating or incorporating reduced-precision effects so it adapts to quantized representations.
- calibration dataconfigured by · Calibration data is data used to measure, estimate, or adjust a model’s parameters, outputs, or behavior for a specified calibration objective.
- SmoothQuantimplemented by · A post-training quantization technique that reduces memory footprint and speeds up inference by smoothing activation and weight ranges.
- 8-bit signed integeruses · An 8-bit signed integer is a fixed-width integer type representing whole-number values from −128 through 127.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.