LLM.int8()
LLM.int8() is a mixed-precision quantization method for large language models that performs most computation in INT8 while handling activation outliers at higher precision.
Quantization and Fast Inference
LLM.int8() is a quantization method for large language models that uses 8-bit computation while handling activation outliers with higher precision.
- How it is used
- Uses mixed-precision quantization to reduce large-language-model memory while preserving quality around activation outliers.
- How deep it goes
- Derives the four-step decomposition, per-channel weight and per-token activation scaling, and threshold tradeoffs.
- Examples
- Uses OPT-6.7B measurements, a threshold sweep, and bitsandbytes deployment configuration.
- What you can do after
- Apply LLM.int8() through Transformers and bitsandbytes using an 8-bit loading option and configurable outlier threshold.
Read how Quantization and Fast Inference explains LLM.int8() →
Domain-Specific Small Language Models
A quantization approach for large language models that uses mixed-precision INT8 computation with special handling for outlier features to preserve accuracy.
- How it is used
- Reduces inference memory by quantizing most model values to INT8 while retaining FP16 computation for outlier features.
- How deep it goes
- Explains absmax and vector-wise quantization through a three-step matrix multiplication workflow.
- Examples
- Compares GPT-2 generation and reports original-versus-quantized perplexity values.
- What you can do after
- Apply it via Hugging Face Transformers, Accelerate, and bitsandbytes to load an 8-bit model on a GPU.
Read how Domain-Specific Small Language Models explains LLM.int8() →
What to know first
- int4 quantizationINT4 quantization represents numerical values using 4-bit integers, reducing storage requirements while limiting numerical precision.
- float16float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- perplexityPerplexity is a language-model metric, typically the exponential of average cross-entropy, that measures how well a model predicts a sequence, with lower values indicating better predictive fit.
Related concepts
- bitsandbytesuses · bitsandbytes is an open-source library providing memory-efficient optimizers and low-precision operations, including quantization, for machine-learning models.
- float16uses · float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
- matrix-matrix multiplicationuses · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- FP32uses · FP32 is a 32-bit single-precision floating-point format for representing approximate real numbers.
- precisionuses · Numerical precision is the fineness with which a numerical representation distinguishes values, often determined by the number of significant digits or bits available.
Questions about LLM.int8()
What is LLM.int8()?
LLM.int8() is a mixed-precision quantization method for large language models that performs most computation in INT8 while handling activation outliers at higher precision.
Which Manning books explain LLM.int8()?
Quantization and Fast Inference and Domain-Specific Small Language Models.
What should I know before learning LLM.int8()?
Manning books that teach LLM.int8() usually introduce int4 quantization, float16, matrix-matrix multiplication and perplexity first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

