AWQ
AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss.
Quantization and Fast Inference
AWQ is a low-bit post-training quantization method that protects important weights or channels to preserve model quality.
- How it is used
- Uses activation-aware scaling to protect salient weight channels in low-bit LLM deployment.
- How deep it goes
- Derives the scaling identity and grid search, then demonstrates group-wise quantization and deployment configuration.
- Examples
- Measures OPT-6.7B reconstruction and perplexity, and compares Qwen2.5-3B-Instruct AWQ and FP16 outputs on an L4.
- What you can do after
- Apply it via AutoAWQ using calibration data and a specified 4-bit group-wise configuration.
CUDA for LLMs
Activation-aware weight quantization is a method that scales weight channels using activation statistics before quantizing them to reduce output error.
- How it is used
- Uses activation-aware rescaling to protect important channels in weight-only model quantization.
- How deep it goes
- Explains the scaling mechanism, shared-step tradeoff, calibration, and conversion-time folding with a short implementation listing.
- Examples
- Works through a three-weight INT8 group and a per-channel conversion-time rescaling listing.
- What you can do after
- Evaluate/understand when channel scaling reduces output error and when a larger group maximum makes quantization coarser.
Also covered in
- Rearchitecting LLMsIn early access (MEAP)
What to know first
- throughputThroughput is the rate at which a system processes or completes work, data, or requests over a specified period.
- data driftData drift is a change in the statistical distribution or characteristics of data relative to a reference, such as data used to train a model.
- float16float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
- latencyLatency is the elapsed time between initiating an operation or request and receiving its corresponding result or observable response.
Related concepts
- activationuses · An activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- activation magnitudeuses · Activation magnitude is the absolute numerical size of a neural-network activation value, considered individually or across features or channels.
- GPTQcontrasts with · GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error.
- per-channel quantizationuses · Per-channel quantization assigns separate quantization parameters, such as scales, to individual channels along a tensor axis.
Questions about AWQ
What is AWQ?
AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss.
Which Manning books explain AWQ?
Quantization and Fast Inference and CUDA for LLMs.
What should I know before learning AWQ?
Manning books that teach AWQ usually introduce throughput, data drift, float16 and latency first.
What is the difference between AWQ and GPTQ?
AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss. GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

