liveBook · Manning

Chapter 8: Pushing into Low Bits Safely

What you will learn in chapter 8 of Quantization and Fast Inference

This introduction explains why some deployment constraints justify looking beyond established integer quantization methods and identifies the low-bit format families the chapter will examine.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why some deployment constraints justify looking beyond established integer quantization methods and identifies the low-bit format families the chapter will examine.

Key terms: INT8/INT4 boundary, uniform integer grids, low-bit floats, FP8, FP4, distribution-matched codebooks

8.1 Decide when to go below eight bits

This section maps three deployment pressures—accelerator throughput, fine-tuning memory, and CPU-only inference—to low-bit format families, while explaining when existing quantization methods remain preferable.

Key terms: binding constraint, sub-8-bit formats, LLM.int8(), GPTQ, AWQ, TurboQuant, FP8, FP4, BF16, tensor cores

8.2 Use FP8 formats and understand kernel caveats

This section explains the precision–range tradeoff between the two FP8 formats, why each suits different tensors, and how their maximum finite values arise.

Key terms: FP8, E4M3, E5M2, exponent, mantissa, binade, forward-pass tensors, backward-pass tensors, weights, activations

8.2.1 Value distribution: why non-uniform spacing matters

This section explains why FP8’s non-uniform spacing can approximate near-zero-heavy tensors better than INT8 at the same bit width.

Key terms: FP8, E4M3, INT8 symmetric, representable values, quantization error

8.2.2 Scaling is mandatory, and granularity matters

This section explains why FP8 activations need scaling and how blockwise scales improve reconstruction fidelity at a storage cost.

Key terms: E4M3, BF16 activations, scale factor, clipping, per-tensor scaling, block scaling, outlier, block-32, reconstruction MSE

8.2.3 Weight reconstruction: FP8 vs. INT8

This section explains why INT8 per-channel reconstructs the tested weights more accurately than FP8 per-tensor, while FP8 remains viable for inference.

Key terms: weights, activations, per-tensor scaling, FP8 E4M3, FP8 E5M2, INT8 per-channel, FP8 per-tensor, reconstruction MSE, end-to-end perplexity

8.2.4 FP8 deployment: 2× compression, zero quality loss

This section reports a weight-only FP8 deployment result and explains why its memory savings do not automatically translate into faster inference.

Key terms: weight-only FP8, E4M3, per-tensor, BF16, perplexity, weight-only INT8, peak TOPS, Transformer Engine, vLLM, TensorRT-LLM

8.3 Use FP4 with blockwise scaling without losing stability

This section introduces the storage, representation, and scaling tradeoffs that make FP4 more demanding than FP8.

Key terms: FP4 E2M1, FP8 E4M3, scale factor, scale-storage overhead

8.3.1 Seven values and a scale factor

This section explains how FP4 E2M1 uses its limited bit patterns and why its non-uniform value spacing differs from symmetric INT4.

Key terms: FP4, E2M1, exponent bias, normal values, subnormal values, mantissa, binade, FP4 E2M1, INT4 symmetric, scale factor

8.3.2 Blockwise scaling: from optional to mandatory

This section explains why FP4 weights need blockwise scaling and how block size and scale format trade reconstruction error against storage cost.

Key terms: FP4, FP8, per-tensor scaling, block scaling, outlier, block size, FP32 scales, MSE, effective bit width, scale overhead

8.3.3 Cross-format comparison: FP4 block-16 versus INT4 group-128

This section compares FP4 and INT4 weight reconstruction error, showing why four-bit results depend on both value spacing and scaling granularity.

Key terms: FP4 block-16, INT4 group-128, weight reconstruction MSE, per-tensor, non-uniform spacing, scaling granularity, INT4 per-tensor, outlier, uniform grid, groupwise scaling

8.3.4 FP4 deployment: the quality cliff and how block scaling recovers

This section shows that block scaling can recover FP4 model quality, while scale storage and hardware support determine its memory and throughput benefits.

Key terms: per-tensor scaling, block-16 scaling, block-32 scaling, perplexity, BF16 baseline, NVFP4, MXFP4, FP8 E4M3, power-of-two scales, E2M1

8.4 Fine-tune with low-bit adapters

This section explains how NF4 storage, compressed scales, and trainable LoRA adapters make fine-tuning a large language model feasible on a consumer GPU.

Key terms: NF4, codebook, normal distribution, quantiles, FP4 E2M1, INT4 symmetric, reconstruction MSE, NF4 block-64, INT4 group-64, FP4 block-16

8.5 Understand ternary quantization and where it fits

This section introduces ternary weights as a compact, simpler-to-compute representation and explains the accuracy risk caused by having only one level near zero.

Key terms: ternary quantization, multiply-accumulate, ternary digits, density gap, NF4, FP4, absmax normalization, scale factor

8.5.1 Post-training ternary quantization fails

The section compares three post-training ternary methods and explains why their improved scaling still leaves substantial weight-reconstruction error.

Key terms: naive per-tensor, TWN per-tensor, TWN per-row, threshold τ, α, reconstruction MSE, NF4 block-64, three representable values

8.5.2 BitLinear: making ternary trainable

Explains how BitLinear trains with ternary weights and uses a controlled comparison to show why training with quantization is more robust than ternarizing afterward.

Key terms: BitNet b1.58, ternary rounding, full-precision master weights, straight-through estimator, STE, subtract-detach-add idiom, round-and-clamp, QAT, PTQ, BitLinear

8.5.3 What you can run today

This section shows what trained ternary models can deliver today, qualifies the evidence for their quality and speed, and ties sub-8-bit format choices to memory–quality constraints.

Key terms: BitNet b1.58 2B4T, OPT-2.7B, WikiText-2 perplexity, bitnet.cpp, ternary PTQ, quantization-aware training, BitLinear, Falcon-Edge, Falcon3 1.58-bit, memory–quality frontier

8.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 7: Quantizing Large Language Models in Practice Chapter 9: Shipping with the Right Toolchain →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.