Chapter 8: Pushing into Low Bits Safely
This introduction explains why some deployment constraints justify looking beyond established integer quantization methods and identifies the low-bit format families the chapter will examine.
After this chapter you will be able to
- Reasons to Go Below Eight Bits. Explain why established integer quantization is sufficient for most production deployments but may not meet unusually tight throughput, memory, or edge-latency constraints. Name the three format families introduced as responses without claiming this introduction establishes which one to choose.
- Constraint Based Format Choice. Explain why the binding deployment constraint, rather than bit width alone, determines whether to move beyond existing INT8 or INT4 methods.
- Accelerator Throughput Formats. Explain why FP8, and conditionally FP4, can address an inference-throughput bottleneck that weight-only INT4 compression does not.
- NF4 Adapter Fine Tuning. Describe how QLoRA uses NF4 to make fine-tuning more accessible on limited GPU memory, and why NF4 differs from INT4 or FP4.
- Ternary CPU Inference Tradeoff. Explain how ternary weights enable a CPU-only inference path and why an existing model cannot simply be rounded to ternary weights.
- FP8 Format Tradeoff. Distinguish E4M3 and E5M2 by how they divide FP8’s exponent and mantissa bits and what that means for precision and range.
- Forward and Backward Format Choice. Justify using E4M3 for forward-pass weights and activations and E5M2 for backward-pass gradients.
- Value spacing and tensor distributions. Explain how the spacing of representable values affects quantization error and why FP8 can outperform symmetric INT8 for tensors concentrated near zero, while INT8 can be better for uniform distributions.
- FP8 activation scale and range. Explain why E4M3 activations must be scaled and describe the scale–quantize–dequantize sequence taught here.
- Blockwise scaling fidelity tradeoff. Distinguish per-tensor from block scaling for FP8 activations, including how outliers affect each, the storage cost, and the observed reconstruction benefit.
- Reconstruction error versus inference quality. Explain why INT8 per-channel has lower weight-reconstruction MSE than FP8 per-tensor despite FP8’s advantage at equal scaling granularity, and distinguish reconstruction MSE from the end-to-end perplexity that matters for inference.
- Weight-only FP8 deployment result. Describe how the OPT-6.7B test stores per-tensor FP8 E4M3 weights, dequantizes them to BF16 for matmul, and checks perplexity on WikiText-2. State that the result is about 2× compression with no perplexity loss.
- FP8 compute kernel gate. Explain the requirements of PyTorch's FP8 matmul primitive and why pre-Hopper GPUs retain FP8 storage savings but not its tensor-core throughput advantage.
- FP4 Compression Tradeoffs. Explain why FP4's storage gain over FP8 makes value placement and scaling more consequential, including the need to account for scale-storage overhead.
- Normal and subnormal spacing. Describe how E2M1 decodes normal and subnormal values and why its gaps grow farther from zero.
- FP4 versus INT4 density. Distinguish FP4 E2M1 from symmetric INT4 by where each places its representable values, and explain why scaling determines whether FP4's near-zero density helps.
- FP4 Block Scaling Necessity. Explain why per-tensor scaling can produce severe error for FP4 weights and why blockwise scaling is needed.
- Block Size Precision Storage Tradeoff. Describe how FP4 block size affects reconstruction error and effective storage, including the block-16 and block-32 comparison.
- FP4 and INT4 comparison. Explain why FP4 block-16 has lower weight reconstruction MSE than INT4 group-128, and why that result does not isolate the effect of format alone. Describe how the per-tensor comparison separates value spacing from scaling granularity.
- Four-bit scaling requirement. Explain why INT4 per-tensor performs poorly in this comparison and why groupwise or blockwise scaling is a practical requirement at four bits.
- FP4 Scaling and Quality. Explain why scaling granularity determines whether FP4 deployment preserves usable model quality in the OPT-6.7B test.
- FP4 Hardware Throughput Caveat. Explain when storing FP4 weights also produces an FP4 compute-speed benefit.
- NF4 distribution-aware codebook. Explain how NF4 places its 16 levels to suit approximately normally distributed weights, and distinguish its lookup table from INT4 uniform spacing and FP4's bit-defined values.
- Double-quantized scale storage. Describe why NF4's block scales consume substantial memory and how double quantization reduces that overhead.
- Frozen-base adapter training. Trace which QLoRA parameters remain frozen, which are trained, and how the quantized base participates in forward and backward passes.
- Consumer-GPU fine-tuning tradeoff. Justify why QLoRA can fit a 7B fine-tuning run on a consumer GPU, while stating the quality and workload limits the section identifies.
- Near-Zero Representation Gap. Explain why ternary quantization risks large weight errors compared with NF4 and FP4, and why changing the scale cannot repair that gap.
- Limits of post-training ternary. Explain why improved ternary PTQ scaling does not adequately reconstruct the weights in this comparison.
- Training with ternary weights. Explain how BitLinear makes ternary weights trainable by quantizing them to {−1, 0, +1} on every forward pass while retaining full-precision master weights for updates.
- Straight-through gradient flow. Describe how the straight-through estimator lets gradients reach full-precision master weights despite round-and-clamp quantization, and identify the subtract-detach-add idiom as its implementation here.
- Ternary Model Deployment Tradeoff. Explain what the BitNet b1.58 2B4T comparison demonstrates about storage, text quality, and CPU inference, while noting the limit of its perplexity comparison with OPT-2.7B.
- Runnable Ternary Model Limits. Distinguish natively trained ternary models from models adapted through quantization-aware fine-tuning and from an untrained throughput demo. Explain why post-training ternary conversion is not presented as a practical path.
What it builds on
- From method comparison to deployment choice (chapter 7) · Assumes familiarity with choosing among existing LLM quantization methods for deployment.
- Apply activation-aware weight protection (AWQ) (chapter 7) · AWQ supplies the weight-only INT4 approach contrasted with changing computation precision.
- Apply Hessian-aware groupwise methods (GPTQ) (chapter 7) · GPTQ is one of the prior weight-quantization methods contrasted with the fine-tuning path.
- Characterize granular and overload errors (chapter 2) · The relationship between representable-value spacing and quantization error.
- Define the affine mapping with scale and zero point (chapter 2) · The idea of using a scale to map values into a quantized range.
- Compare symmetric and asymmetric choices (chapter 2) · Symmetric integer quantization and its scale factor.
- Quantize weights with per-tensor and per-channel choices (chapter 3) · Per-tensor and per-channel weight quantization choices
- Train with fake quantization and straight-through estimators (chapter 5) · Training with fake quantization and straight-through estimators.
Section by section
Introduction
This introduction explains why some deployment constraints justify looking beyond established integer quantization methods and identifies the low-bit format families the chapter will examine.
- Reasons to Go Below Eight Bits: Explain why established integer quantization is sufficient for most production deployments but may not meet unusually tight throughput, memory, or edge-latency constraints. Name the three format families introduced as responses without claiming this introduction establishes which one to choose.
Why it matters: The reason to explore lower-bit formats is a specific deployment constraint, not an assumption that existing integer methods are inadequate.
Key terms: INT8/INT4 boundary, uniform integer grids, low-bit floats, FP8, FP4, distribution-matched codebooks
8.1 Decide when to go below eight bits
This section maps three deployment pressures—accelerator throughput, fine-tuning memory, and CPU-only inference—to low-bit format families, while explaining when existing quantization methods remain preferable.
- Constraint Based Format Choice: Explain why the binding deployment constraint, rather than bit width alone, determines whether to move beyond existing INT8 or INT4 methods.
Why it matters: Choosing by bottleneck avoids adopting a lower-bit format that does not solve the deployment problem. - Accelerator Throughput Formats: Explain why FP8, and conditionally FP4, can address an inference-throughput bottleneck that weight-only INT4 compression does not.
Why it matters: Smaller weights alone need not increase tokens per second when the matrix multiplication still runs at higher precision. - NF4 Adapter Fine Tuning: Describe how QLoRA uses NF4 to make fine-tuning more accessible on limited GPU memory, and why NF4 differs from INT4 or FP4.
Why it matters: It distinguishes a memory-saving fine-tuning path from a read-only quantized artifact or a faster inference format. - Ternary CPU Inference Tradeoff: Explain how ternary weights enable a CPU-only inference path and why an existing model cannot simply be rounded to ternary weights.
Why it matters: Ternary inference offers a route without a GPU, but its training requirement rules out a simple conversion of an existing model.
Key terms: binding constraint, sub-8-bit formats, LLM.int8(), GPTQ, AWQ, TurboQuant, FP8, FP4, BF16, tensor cores
8.2 Use FP8 formats and understand kernel caveats
This section explains the precision–range tradeoff between the two FP8 formats, why each suits different tensors, and how their maximum finite values arise.
- FP8 Format Tradeoff: Distinguish E4M3 and E5M2 by how they divide FP8’s exponent and mantissa bits and what that means for precision and range.
Why it matters: The bit split determines whether an FP8 tensor gets finer rounding or more room for large values. - Forward and Backward Format Choice: Justify using E4M3 for forward-pass weights and activations and E5M2 for backward-pass gradients.
Why it matters: Choosing the format by tensor role avoids spending precision or range where it provides less benefit. - Maximum Finite Value Derivation: Explain how the exponent, mantissa, and reserved patterns set the maximum finite values of E4M3 and E5M2.
Why it matters: The derivation shows why the formats’ range limits differ rather than treating those limits as arbitrary numbers.
Key terms: FP8, E4M3, E5M2, exponent, mantissa, binade, forward-pass tensors, backward-pass tensors, weights, activations
8.2.1 Value distribution: why non-uniform spacing matters
This section explains why FP8’s non-uniform spacing can approximate near-zero-heavy tensors better than INT8 at the same bit width.
- Value spacing and tensor distributions: Explain how the spacing of representable values affects quantization error and why FP8 can outperform symmetric INT8 for tensors concentrated near zero, while INT8 can be better for uniform distributions.
Why it matters: The tensor’s value distribution, not bit width alone, helps determine which format approximates it better.
Key terms: FP8, E4M3, INT8 symmetric, representable values, quantization error
8.2.2 Scaling is mandatory, and granularity matters
This section explains why FP8 activations need scaling and how blockwise scales improve reconstruction fidelity at a storage cost.
- FP8 activation scale and range: Explain why E4M3 activations must be scaled and describe the scale–quantize–dequantize sequence taught here.
Why it matters: Without scaling, FP8 can discard activation information before the choice of scaling granularity even matters. - Blockwise scaling fidelity tradeoff: Distinguish per-tensor from block scaling for FP8 activations, including how outliers affect each, the storage cost, and the observed reconstruction benefit.
Why it matters: Scaling granularity determines whether a few outliers degrade reconstruction across a whole activation tensor or only within their blocks.
Key terms: E4M3, BF16 activations, scale factor, clipping, per-tensor scaling, block scaling, outlier, block-32, reconstruction MSE
8.2.3 Weight reconstruction: FP8 vs. INT8
This section explains why INT8 per-channel reconstructs the tested weights more accurately than FP8 per-tensor, while FP8 remains viable for inference.
- Weight scaling suitability: Explain why per-tensor scaling can suffice for weights in this comparison: unlike activations, weights have no dynamic outliers or batch variation.
Why it matters: This explains why the FP8 weight comparison uses one scale for the whole matrix. - Reconstruction error versus inference quality: Explain why INT8 per-channel has lower weight-reconstruction MSE than FP8 per-tensor despite FP8’s advantage at equal scaling granularity, and distinguish reconstruction MSE from the end-to-end perplexity that matters for inference.
Why it matters: A lower weight-reconstruction error alone does not establish which format gives better inference quality.
Key terms: weights, activations, per-tensor scaling, FP8 E4M3, FP8 E5M2, INT8 per-channel, FP8 per-tensor, reconstruction MSE, end-to-end perplexity
8.2.4 FP8 deployment: 2× compression, zero quality loss
This section reports a weight-only FP8 deployment result and explains why its memory savings do not automatically translate into faster inference.
- Weight-only FP8 deployment result: Describe how the OPT-6.7B test stores per-tensor FP8 E4M3 weights, dequantizes them to BF16 for matmul, and checks perplexity on WikiText-2. State that the result is about 2× compression with no perplexity loss.
Why it matters: It separates the demonstrated storage and quality result from a claim of faster FP8 computation. - FP8 and INT8 deployment choice: Explain why the section favors FP8 as a production path on Hopper and later hardware despite its weight-only result being nearly identical to INT8.
Why it matters: A deployment format can be attractive because the software stack supports it, even when its measured compression and peak hardware throughput match an alternative. - FP8 compute kernel gate: Explain the requirements of PyTorch's FP8 matmul primitive and why pre-Hopper GPUs retain FP8 storage savings but not its tensor-core throughput advantage.
Why it matters: It prevents a smaller FP8 model from being mistaken for automatically faster FP8 inference.
Key terms: weight-only FP8, E4M3, per-tensor, BF16, perplexity, weight-only INT8, peak TOPS, Transformer Engine, vLLM, TensorRT-LLM
8.3 Use FP4 with blockwise scaling without losing stability
This section introduces the storage, representation, and scaling tradeoffs that make FP4 more demanding than FP8.
- FP4 Compression Tradeoffs: Explain why FP4's storage gain over FP8 makes value placement and scaling more consequential, including the need to account for scale-storage overhead.
Why it matters: The smaller weight representation is useful only if its limited values and scaling costs still meet the model's needs.
Key terms: FP4 E2M1, FP8 E4M3, scale factor, scale-storage overhead
8.3.1 Seven values and a scale factor
This section explains how FP4 E2M1 uses its limited bit patterns and why its non-uniform value spacing differs from symmetric INT4.
- E2M1 finite value budget: Explain how FP4 E2M1 allocates its four bits and why it reserves no patterns for infinity or NaN.
Why it matters: The small bit budget explains why every representable value matters in FP4. - Normal and subnormal spacing: Describe how E2M1 decodes normal and subnormal values and why its gaps grow farther from zero.
Why it matters: The decoding rules explain the uneven spacing shown in the FP4 value table. - FP4 versus INT4 density: Distinguish FP4 E2M1 from symmetric INT4 by where each places its representable values, and explain why scaling determines whether FP4's near-zero density helps.
Why it matters: The comparison shows both FP4's potential advantage and why that advantage is not automatic.
Key terms: FP4, E2M1, exponent bias, normal values, subnormal values, mantissa, binade, FP4 E2M1, INT4 symmetric, scale factor
8.3.2 Blockwise scaling: from optional to mandatory
This section explains why FP4 weights need blockwise scaling and how block size and scale format trade reconstruction error against storage cost.
- FP4 Block Scaling Necessity: Explain why per-tensor scaling can produce severe error for FP4 weights and why blockwise scaling is needed.
Why it matters: This explains why a scaling choice that worked for FP8 weights is unsafe for FP4. - Block Size Precision Storage Tradeoff: Describe how FP4 block size affects reconstruction error and effective storage, including the block-16 and block-32 comparison.
Why it matters: The smallest error is not automatically the best choice when scale storage erodes compression. - NVFP4 Two Level Scaling: Explain how NVFP4 reduces block-scale overhead while retaining block-16 scaling.
Why it matters: It shows how the choice of scale format can preserve fine-grained scaling without paying the full FP32-scale cost.
Key terms: FP4, FP8, per-tensor scaling, block scaling, outlier, block size, FP32 scales, MSE, effective bit width, scale overhead
8.3.3 Cross-format comparison: FP4 block-16 versus INT4 group-128
This section compares FP4 and INT4 weight reconstruction error, showing why four-bit results depend on both value spacing and scaling granularity.
- FP4 and INT4 comparison: Explain why FP4 block-16 has lower weight reconstruction MSE than INT4 group-128, and why that result does not isolate the effect of format alone. Describe how the per-tensor comparison separates value spacing from scaling granularity.
Why it matters: It prevents you from attributing an error improvement to FP4 when the configurations also use different numbers of scales. - Four-bit scaling requirement: Explain why INT4 per-tensor performs poorly in this comparison and why groupwise or blockwise scaling is a practical requirement at four bits.
Why it matters: It explains why choosing a four-bit format without sufficiently local scaling can produce large reconstruction errors.
Key terms: FP4 block-16, INT4 group-128, weight reconstruction MSE, per-tensor, non-uniform spacing, scaling granularity, INT4 per-tensor, outlier, uniform grid, groupwise scaling
8.3.4 FP4 deployment: the quality cliff and how block scaling recovers
This section shows that block scaling can recover FP4 model quality, while scale storage and hardware support determine its memory and throughput benefits.
- FP4 Scaling and Quality: Explain why scaling granularity determines whether FP4 deployment preserves usable model quality in the OPT-6.7B test.
Why it matters: Choosing four-bit weights is not enough; the scaling choice can determine whether the deployed model remains useful. - Four Bit Scale Storage: Distinguish NVFP4 and MXFP4 by their scale storage and the resulting size-versus-precision tradeoff.
Why it matters: The scale metadata affects both the actual memory footprint and the accuracy of a four-bit format. - FP4 Hardware Throughput Caveat: Explain when storing FP4 weights also produces an FP4 compute-speed benefit.
Why it matters: A smaller checkpoint does not by itself guarantee faster inference. - Simulated FP4 Quality Measurement: Explain why the companion script can measure FP4 quality without running native FP4 kernels.
Why it matters: It lets you interpret the reported quality measurements without mistaking the simulation for a native-kernel speed test.
Key terms: per-tensor scaling, block-16 scaling, block-32 scaling, perplexity, BF16 baseline, NVFP4, MXFP4, FP8 E4M3, power-of-two scales, E2M1
8.4 Fine-tune with low-bit adapters
This section explains how NF4 storage, compressed scales, and trainable LoRA adapters make fine-tuning a large language model feasible on a consumer GPU.
- NF4 distribution-aware codebook: Explain how NF4 places its 16 levels to suit approximately normally distributed weights, and distinguish its lookup table from INT4 uniform spacing and FP4's bit-defined values.
Why it matters: The codebook explains what NF4 stores and why its levels suit typical neural-network weights. - Matched-granularity reconstruction comparison: Explain what the weight-reconstruction comparisons show when scaling granularity is held equal, and why FP4's similar measured MSE does not establish a better codebook.
Why it matters: Comparing formats fairly prevents a finer scaling scheme from being mistaken for a better codebook. - Double-quantized scale storage: Describe why NF4's block scales consume substantial memory and how double quantization reduces that overhead.
Why it matters: Compressing the scales helps the complete fine-tuning run fit within limited GPU memory. - Frozen-base adapter training: Trace which QLoRA parameters remain frozen, which are trained, and how the quantized base participates in forward and backward passes.
Why it matters: This separation lets a compressed model adapt to a task without training all of its weights. - Consumer-GPU fine-tuning tradeoff: Justify why QLoRA can fit a 7B fine-tuning run on a consumer GPU, while stating the quality and workload limits the section identifies.
Why it matters: The memory savings make domain adaptation accessible on smaller hardware, without making quality or memory limits disappear.
Key terms: NF4, codebook, normal distribution, quantiles, FP4 E2M1, INT4 symmetric, reconstruction MSE, NF4 block-64, INT4 group-64, FP4 block-16
8.5 Understand ternary quantization and where it fits
This section introduces ternary weights as a compact, simpler-to-compute representation and explains the accuracy risk caused by having only one level near zero.
- Ternary Values and Storage: Describe ternary weights as −1, 0, or +1, and explain how those three choices simplify arithmetic and allow compact storage.
Why it matters: The storage and arithmetic savings explain why ternary weights are attractive despite their accuracy risk. - Near-Zero Representation Gap: Explain why ternary quantization risks large weight errors compared with NF4 and FP4, and why changing the scale cannot repair that gap.
Why it matters: This representation gap explains why ternary's substantial compression does not by itself guarantee a usable model.
Key terms: ternary quantization, multiply-accumulate, ternary digits, density gap, NF4, FP4, absmax normalization, scale factor
8.5.1 Post-training ternary quantization fails
The section compares three post-training ternary methods and explains why their improved scaling still leaves substantial weight-reconstruction error.
- Ternary PTQ method differences: Describe how naive per-tensor quantization, TWN per-tensor, and TWN per-row assign weights to ternary values.
Why it matters: These differences show what the progressively stronger PTQ methods change before their errors are compared. - Limits of post-training ternary: Explain why improved ternary PTQ scaling does not adequately reconstruct the weights in this comparison.
Why it matters: It explains why finer scaling alone cannot make post-training ternary weights reconstruct well.
Key terms: naive per-tensor, TWN per-tensor, TWN per-row, threshold τ, α, reconstruction MSE, NF4 block-64, three representable values
8.5.2 BitLinear: making ternary trainable
Explains how BitLinear trains with ternary weights and uses a controlled comparison to show why training with quantization is more robust than ternarizing afterward.
- Training with ternary weights: Explain how BitLinear makes ternary weights trainable by quantizing them to {−1, 0, +1} on every forward pass while retaining full-precision master weights for updates.
Why it matters: This is why ternary quantization can work better when it is part of training rather than imposed afterward. - Straight-through gradient flow: Describe how the straight-through estimator lets gradients reach full-precision master weights despite round-and-clamp quantization, and identify the subtract-detach-add idiom as its implementation here.
Why it matters: Without gradient flow through quantization, the master weights could not learn from the ternary forward pass. - QAT versus post-training comparison: Describe how the toy MLP comparison isolates the training method and what its reported QAT-versus-PTQ gap illustrates.
Why it matters: The comparison provides evidence that adaptation during training, rather than the architecture or data, explains the advantage shown here.
Key terms: BitNet b1.58, ternary rounding, full-precision master weights, straight-through estimator, STE, subtract-detach-add idiom, round-and-clamp, QAT, PTQ, BitLinear
8.5.3 What you can run today
This section shows what trained ternary models can deliver today, qualifies the evidence for their quality and speed, and ties sub-8-bit format choices to memory–quality constraints.
- Ternary Model Deployment Tradeoff: Explain what the BitNet b1.58 2B4T comparison demonstrates about storage, text quality, and CPU inference, while noting the limit of its perplexity comparison with OPT-2.7B.
Why it matters: The example shows a substantial storage benefit without treating an imperfect quality comparison as an exact match. - Runnable Ternary Model Limits: Distinguish natively trained ternary models from models adapted through quantization-aware fine-tuning and from an untrained throughput demo. Explain why post-training ternary conversion is not presented as a practical path.
Why it matters: Knowing how a model was trained keeps a speed demonstration or an adapted checkpoint from being mistaken for proof of native ternary quality. - Constraint Driven Format Choice: Explain how the memory–quality frontier frames the choice among sub-8-bit formats and state the practical CPU alternative when ternary training is unavailable.
Why it matters: A format choice should reflect the memory and quality limits you must meet, as well as the training resources you actually have.
Key terms: BitNet b1.58 2B4T, OPT-2.7B, WikiText-2 perplexity, bitnet.cpp, ternary PTQ, quantization-aware training, BitLinear, Falcon-Edge, Falcon3 1.58-bit, memory–quality frontier
8.6 Summary
Where this is used later
- Choose graphs, runtimes, and artifacts for deployment (chapter 9) · Choosing deployment graphs and runtimes depends on identifying the constraint the format must address.
- Build TensorRT engines with real integer execution (chapter 9) · Informs the distinction between a compressed artifact and execution on a supported low-precision kernel.
- Understand the GGUF format and its variants (chapter 10) · GGUF is the packaging named for the practical CPU alternative.
Concepts explained in this chapter
- NormalFloat4NF4 is a 4-bit NormalFloat format that provides nonuniform representable levels suited to normally distributed neural-network weights.
- FP8A family of 8-bit floating-point formats for reduced-precision numerical computation.
- E4M3An 8-bit floating-point format with one sign bit, four exponent bits, and three mantissa bits.
- FP4A family of 4-bit floating-point formats for reduced-precision numerical representation.
- block quantizationA technique that assigns scaling parameters separately to fixed-size blocks of numerical values rather than to an entire tensor.
- ternary modelA neural-network model whose weights use three discrete representable values.
- per-tensor quantizationPer-tensor quantization uses one set of quantization parameters for all values in a tensor.
- QLoRAQLoRA is a parameter-efficient adaptation method that fine-tunes a quantized base model with low-rank adapters.
- double quantizationDouble quantization quantizes quantization parameters, such as block scales, to reduce their storage cost.
- low-rank adaptationLoRA is a parameter-efficient fine-tuning method that freezes pretrained weights and learns low-rank update matrices.
- BitLinearBitLinear is the example's linear-layer implementation that quantizes weights and activations during training while preserving gradient flow.
- ternary quantizationA technique that represents numerical values using three levels, commonly negative one, zero, and positive one.
- granularityThe level of tensor partitioning that determines which values share quantization parameters such as a scale factor.
- BitNet b1.58 2B4TBitNet b1.58 2B4T is a 2-billion-parameter language model trained on 4 trillion tokens with ternary weights.
- quantization-aware trainingQuantization-aware training incorporates simulated quantization effects while optimizing a model so it can retain accuracy at deployment precision.
- NVFP4NVFP4 is NVIDIA's 4-bit floating-point format that stores block scales in FP8 and a tensor-wide scale in FP32.
- 8-bit signed integerINT8 is an 8-bit signed or unsigned integer representation used for low-precision numerical computation and storage.
- straight-through estimatorA straight-through estimator approximates gradients through a nondifferentiable operation using a surrogate derivative.
- precisionThe numerical detail available to represent a value, often determined by the number of bits used.
- resolutionResolution is the spacing between adjacent representable numerical values.
- mean squared errorMean squared error is the average squared difference between corresponding numerical values.
- per-channel quantizationPer-channel quantization assigns separate quantization parameters to each tensor channel.
- perplexityPerplexity is a language-model metric measuring how well a probability model predicts a sequence, with lower values indicating better predictive performance.
- normal distributionA normal distribution is a continuous probability distribution that is symmetric about its mean and has a bell-shaped density.
- percentileA quantile is a value that divides a probability distribution so that a specified proportion of observations falls below it.
- int4 quantizationA 4-bit integer representation used for low-precision numerical computation and storage.
- quantization gridA quantization grid is a discrete set of representable numerical levels with a defined spacing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
