llm_int8_threshold sets the activation-magnitude cutoff for LLM.int8() outlier detection. Dimensions above the cutoff use the full-precision path; the rest use INT8. A value of 6.0 is well-calibrated for the layers where tuning matters most. section 7.2
Why the cutoff matters
In an OPT-6.7B middle layer, threshold 6.0 flags 0.2% of dimensions and reduces error 172×. In late layer 31, it flags 1% and reduces error 14×; middle layers show the clearest tuning knee. section 7.2
Figure 7.6 Threshold sweep for OPT-6.7B. Left: fraction of dimensions routed to FP16 as a function of threshold. Right: error reduction ratio (log scale). The shaded band marks the 5.0-7.0 sweet spot. Each layer reveals a different personality: layer 0 (circles, solid) is threshold-insensitive, layer 16 (squares, dashed) shows the clearest knee at 6.0, and layer 31 (triangles, dash-dot) has broadly elevated activations requiring more outlier routing at any threshold.

Compare how threshold changes affect outlier routing and error across OPT-6.7B layers; the sweep marks 5.0–7.0 as the sweet spot. section 7.2
Listing 7.5 LLM.int8() deployment with bitsandbytes
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(
load_in_8bit=True, #A
llm_int8_threshold=6.0, #B
)
model = AutoModelForCausalLM.from_pretrained(
"facebook/opt-6.7b",
quantization_config=quantization_config,
device_map="auto",
)
See the Hugging Face Transformers configuration that sets llm_int8_threshold to 6.0 for bitsandbytes. section 7.2
