liveBook · Manning

AWQ

AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss.

How 2 Manning books explain it

Quantization and Fast Inference

AWQ is a low-bit post-training quantization method that protects important weights or channels to preserve model quality.

How it is used
Uses activation-aware scaling to protect salient weight channels in low-bit LLM deployment.
How deep it goes
Derives the scaling identity and grid search, then demonstrates group-wise quantization and deployment configuration.
Examples
Measures OPT-6.7B reconstruction and perplexity, and compares Qwen2.5-3B-Instruct AWQ and FP16 outputs on an L4.
What you can do after
Apply it via AutoAWQ using calibration data and a specified 4-bit group-wise configuration.

Read how Quantization and Fast Inference explains AWQ →

CUDA for LLMs

Activation-aware weight quantization is a method that scales weight channels using activation statistics before quantizing them to reduce output error.

How it is used
Uses activation-aware rescaling to protect important channels in weight-only model quantization.
How deep it goes
Explains the scaling mechanism, shared-step tradeoff, calibration, and conversion-time folding with a short implementation listing.
Examples
Works through a three-weight INT8 group and a per-channel conversion-time rescaling listing.
What you can do after
Evaluate/understand when channel scaling reduces output error and when a larger group maximum makes quantization coarser.

Read how CUDA for LLMs explains AWQ →

Also covered in

What to know first

Related concepts

Questions about AWQ

What is AWQ?

AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss.

Which Manning books explain AWQ?

Quantization and Fast Inference and CUDA for LLMs.

What should I know before learning AWQ?

Manning books that teach AWQ usually introduce throughput, data drift, float16 and latency first.

What is the difference between AWQ and GPTQ?

AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss. GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.