liveBook · Manning

LLM.int8()

LLM.int8() is a mixed-precision quantization method for large language models that performs most computation in INT8 while handling activation outliers at higher precision.

How 2 Manning books explain it

Quantization and Fast Inference

LLM.int8() is a quantization method for large language models that uses 8-bit computation while handling activation outliers with higher precision.

How it is used
Uses mixed-precision quantization to reduce large-language-model memory while preserving quality around activation outliers.
How deep it goes
Derives the four-step decomposition, per-channel weight and per-token activation scaling, and threshold tradeoffs.
Examples
Uses OPT-6.7B measurements, a threshold sweep, and bitsandbytes deployment configuration.
What you can do after
Apply LLM.int8() through Transformers and bitsandbytes using an 8-bit loading option and configurable outlier threshold.

Read how Quantization and Fast Inference explains LLM.int8() →

Domain-Specific Small Language Models

A quantization approach for large language models that uses mixed-precision INT8 computation with special handling for outlier features to preserve accuracy.

How it is used
Reduces inference memory by quantizing most model values to INT8 while retaining FP16 computation for outlier features.
How deep it goes
Explains absmax and vector-wise quantization through a three-step matrix multiplication workflow.
Examples
Compares GPT-2 generation and reports original-versus-quantized perplexity values.
What you can do after
Apply it via Hugging Face Transformers, Accelerate, and bitsandbytes to load an 8-bit model on a GPU.

Read how Domain-Specific Small Language Models explains LLM.int8() →

What to know first

Related concepts

Questions about LLM.int8()

What is LLM.int8()?

LLM.int8() is a mixed-precision quantization method for large language models that performs most computation in INT8 while handling activation outliers at higher precision.

Which Manning books explain LLM.int8()?

Quantization and Fast Inference and Domain-Specific Small Language Models.

What should I know before learning LLM.int8()?

Manning books that teach LLM.int8() usually introduce int4 quantization, float16, matrix-matrix multiplication and perplexity first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.