LLM.int8() quantizes large language models with mixed precision: ordinary values use INT8, while unusually large outlier features remain in FP16 to preserve accuracy. It reduces inference memory by roughly half without requiring a custom implementation. section 6.2.2
The computation proceeds in three stages:
- Extract columns containing outlier features from the hidden states with a threshold. section 6.2.2
- Multiply outliers in FP16 and remaining values in INT8 using vector-wise absmax quantization. section 6.2.2
- Dequantize the INT8 results to FP16, add the outlier results, and produce the full FP16 output. section 6.2.2
The approach targets models above roughly seven billion parameters, where outliers can make uniform quantization inaccurate. It runs on GPUs through bitsandbytes and integrates with Hugging Face Transformers via Accelerate. section 6.2.2 section 9.2
GPT-2 loading and evaluation
Load GPT-2 with device_map='auto' and load_in_8bit=True, generate text from the prompt “My favourite school subject is,” and compare perplexity. The reported perplexities are 11.25 for the original model and 15.07 for LLM.int8().
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model_id = 'openai-community/gpt2'
model_int8 = AutoModelForCausalLM.from_pretrained(model_id,
device_map='auto',
load_in_8bit=True,
)
Shows the minimal Transformers configuration for loading a model with LLM.int8() quantization. section 6.2.2
