liveBook · Manning

Llm_int8_threshold

As explained in Quantization and Fast Inference

llm_int8_threshold is a configuration parameter that sets the activation-magnitude threshold for identifying outlier dimensions in LLM.int8().

llm_int8_threshold sets the activation-magnitude cutoff for LLM.int8() outlier detection. Dimensions above the cutoff use the full-precision path; the rest use INT8. A value of 6.0 is well-calibrated for the layers where tuning matters most. section 7.2

Why the cutoff matters

In an OPT-6.7B middle layer, threshold 6.0 flags 0.2% of dimensions and reduces error 172×. In late layer 31, it flags 1% and reduces error 14×; middle layers show the clearest tuning knee. section 7.2

Figure 7.6 Threshold sweep for OPT-6.7B. Left: fraction of dimensions routed to FP16 as a function of threshold. Right: error reduction ratio (log scale). The shaded band marks the 5.0-7.0 sweet spot. Each layer reveals a different personality: layer 0 (circles, solid) is threshold-insensitive, layer 16 (squares, dashed) shows the clearest knee at 6.0, and layer 31 (triangles, dash-dot) has broadly elevated activations requiring more outlier routing at any threshold.

Compare how threshold changes affect outlier routing and error across OPT-6.7B layers; the sweep marks 5.0–7.0 as the sweet spot. section 7.2

Listing 7.5 LLM.int8() deployment with bitsandbytes
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
 
quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,                                            #A
    llm_int8_threshold=6.0,                                       #B
)
 
model = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-6.7b",
    quantization_config=quantization_config,
    device_map="auto",
)
 

See the Hugging Face Transformers configuration that sets llm_int8_threshold to 6.0 for bitsandbytes. section 7.2

How Quantization and Fast Inference covers it

How deep it goes
Defines an activation-magnitude cutoff and compares its layer-wise effects, recommending 6.0 without model-specific calibration.
Examples
Compares threshold effects across OPT-6.7B layers, including a middle-layer knee and 1% outlier routing in layer 31.
What you can do after
Apply it via bitsandbytes' BitsAndBytesConfig by setting llm_int8_threshold, with 6.0 as the calibrated default.
How it connects
Connects activation outliers, mixed-precision matrix multiplication, INT8 quantization, and bitsandbytes.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.