Chapter 5: Multi-token prediction and FP8 quantization
This chapter introduction shifts from DeepSeek’s architecture to its training techniques and distinguishes FP8 use in training from its earlier use in inference.
After this chapter you will be able to
- Architecture and training roles. Distinguish the roles assigned here to Multi-Head Latent Attention and Mixture-of-Experts versus Multi-Token Prediction and FP8 quantization.
- FP8 inference and training. Explain what the introduction identifies as distinctive about DeepSeek’s use of FP8 quantization.
- Multi-token training objective. Distinguish standard next-token prediction from multi-token prediction (MTP), including what the loss evaluates at a given position.
- Denser multi-token training signals. Explain how MTP gives a token position feedback about several upcoming tokens rather than only the next one, and why the section says this helps the model learn longer-range relationships from a training example.
- Data efficiency from denser signals. Explain why MTP can achieve better performance with the same amount of training data: predicting multiple future tokens makes each training sample more informative.
- Choice points in MTP loss. Explain what a choice point is and trace how predicting multiple future tokens makes its error recur in the loss, encouraging the model to handle consequential transitions better.
- Speculative decoding workflow. Explain how speculative decoding replaces repeated single-token generation with drafting several candidate tokens and verifying them together, and why MTP heads can supply the draft.
- Sequential MTP prediction chain. Explain how DeepSeek’s sequential MTP modules differ from independent output heads: each module builds on the previous module’s hidden state, so the representation used for token t+1 informs the prediction of t+2. Clarify that training uses ground-truth tokens through teacher forcing, not the model’s sampled predictions.
- Shared trunk hidden states. Trace how the input sequence passes through the shared Transformer trunk to produce h⁰, and explain why those hidden states—not raw tokens—start the MTP process.
- Causal MTP module chain. Explain how DeepSeek-V3 uses one MTP module per prediction depth and passes each module's refined hidden state to the next rather than using independent prediction heads.
- MTP head input construction. Trace how a head takes the previous depth's hidden state and, during training, the ground-truth future-token embedding, then normalizes, concatenates, and projects them into its transformer input.
- Refined state and token logits. Describe how the dedicated transformer block produces a new hidden state and how that state both feeds the next MTP module and produces future-token logits through the shared output head.
- Summed multi-token training loss. Explain how training calculates the loss for one input token across D prediction depths and why that gives the model a broader training signal.
- MTP module and model organization. Describe how a dedicated MTP module fits into the full model: it refines a previous hidden state using a future-token embedding, while the model provides one module per prediction depth and shares its embedding and output layers.
- Causal forward pass and loss. Trace how the trunk's hidden state passes through successive MTP modules to produce future-token logits, and describe how their losses join the main next-token loss. Explain what the shrinking logit sequence lengths check.
- Low-precision efficiency trade-off. Explain why the section presents FP8 quantization as a complement to MLA, MoE, and MTP: accepting lower numerical precision can improve speed and memory efficiency, helping make large-scale training affordable.
- Quantization and precision trade-off. Explain how quantization converts model parameters to a lower bit-width, using FP32 to FP16 as an example, and why a model may still perform well despite losing some numerical precision.
- Quantization memory and speed tradeoff. Explain how reducing parameter precision saves memory, why that can speed training and inference and make large models more accessible, and what is traded away.
- Floating-point bit roles. Explain what the sign, exponent, and mantissa contribute to a floating-point number, and how exponent and mantissa bits affect range and precision.
- FP32 FP16 BF16 trade-offs. Distinguish FP32, FP16, and BF16 by their exponent and mantissa allocations, and explain the resulting range and precision trade-offs.
- INT8 and FP8 distinction. Distinguish INT8’s uniformly spaced integer values from FP8’s floating-point representation, and explain the limitation of INT8 when values span widely different magnitudes.
- Tensor scaling for INT8 quantization. Explain how to find a tensor’s maximum absolute value, use it to scale FP32 values into the −127 to 127 range, round them to integers, and approximately recover them by dividing by the same factor.
- Mixed precision trade-off. Explain why DeepSeek assigns different numerical formats to different parts of training rather than using FP8 or FP32 everywhere.
- Forward and input-gradient flow. Trace the precision choices for the linear layer’s forward pass and its backward input-gradient calculation.
- Weight gradients and updates. Distinguish the precision used to calculate the weight gradient from the precision used to store it and update the master weights.
- Outlier sensitivity of tensor scaling. Explain why one scale for an entire tensor can erase distinctions among smaller values when the tensor contains an outlier.
- Independent activation and weight scales. Describe how fine-grained quantization partitions activations and weights and why each partition receives its own scale.
- Low precision accumulation problem. Explain why accumulating a long FP8 dot product entirely in a Tensor Core can produce numerical error, even though the hardware performs FP8 multiplication quickly.
- Periodic FP32 promotion. Trace how DeepSeek divides accumulation between Tensor Cores and CUDA Cores, including when partial sums are transferred and where the running total is kept.
- Uniform E4M3 through local scaling. Explain how fine-grained quantization lets DeepSeek use E4M3 in both forward and backward passes despite its smaller dynamic range.
- Online scaling workflow. Describe how online quantization derives and applies a scaling factor for the current tensor, and explain its computational trade-off.
What it builds on
- MLA: The best of both worlds (chapter 3) · Introduces Multi-Head Latent Attention as an architectural approach.
- The intuition behind mixture of experts (chapter 4) · Introduces Mixture-of-Experts as an architectural approach.
- The core task: Predicting the next token (chapter 2) · The standard task of predicting the next token.
- Autoregressive process: Appending tokens to build context (chapter 2) · Standard autoregressive generation proceeds one token at a time.
- From input embeddings to context vectors: A mathematical walkthrough (chapter 2) · The idea of context vectors produced from input tokens.
Section by section
Introduction
This chapter introduction shifts from DeepSeek’s architecture to its training techniques and distinguishes FP8 use in training from its earlier use in inference.
- Architecture and training roles: Distinguish the roles assigned here to Multi-Head Latent Attention and Mixture-of-Experts versus Multi-Token Prediction and FP8 quantization.
Why it matters: This distinction frames why the chapter moves from model architecture to training methodology. - FP8 inference and training: Explain what the introduction identifies as distinctive about DeepSeek’s use of FP8 quantization.
Why it matters: It identifies the training application—not the existence of FP8 inference—as the claim this chapter builds on.
Key terms: Multi-Head Latent Attention, Mixture-of-Experts, Multi-Token Prediction, FP8 Quantization, inference, large-scale training
5.1 The core idea: From single-token to multi-token prediction
This section contrasts standard next-token training with multi-token prediction and briefly explains how extra predictions can also speed up inference.
- Multi-token training objective: Distinguish standard next-token prediction from multi-token prediction (MTP), including what the loss evaluates at a given position.
Why it matters: The prediction horizon is the central change that distinguishes MTP from the training objective used earlier in the book. - Faster decoding opportunity: Explain the limited inference-time benefit described for MTP's extra prediction heads.
Why it matters: MTP is presented as useful not only for its training signal but also for potentially faster generation.
Key terms: Next-Token Prediction, Multi-Token Prediction, loss, speculative decoding, extra prediction heads, verification step
5.2 The four key advantages of MTP
Introduces the claim that predicting multiple tokens during training changes what and how efficiently the model learns, ahead of a discussion of four benefits.
5.2.1 Densification of training signals
Explains why predicting several future tokens from each position gives MTP denser training feedback than single-token prediction.
- Denser multi-token training signals: Explain how MTP gives a token position feedback about several upcoming tokens rather than only the next one, and why the section says this helps the model learn longer-range relationships from a training example.
Why it matters: This is the section’s central reason that MTP offers a richer learning signal per training example.
Key terms: MTP, single-token prediction, training signals, gradient signal
5.2.2 Improved data efficiency
This section explains how multi-token prediction (MTP) improves data efficiency and summarizes benchmark evidence, including limits to its gains.
- Data efficiency from denser signals: Explain why MTP can achieve better performance with the same amount of training data: predicting multiple future tokens makes each training sample more informative.
Why it matters: Training data can be expensive, so learning more from the same data is valuable. - Benchmark trends and limits: Describe the reported coding-benchmark trends: MTP's advantage over single-token prediction becomes more consistent as model size increases, while predicting more future tokens generally helps with fixed training data only up to a point.
Why it matters: These results support the data-efficiency claim without implying that MTP wins at every scale or that more predictions always help.
Key terms: MTP, data efficiency, densification of training signals, single-token baseline, predicted future tokens, n, MBPP, HumanEval
5.2.3 Better planning by prioritizing "choice points"
Explains how multi-token prediction can improve planning by making consequential pattern-shifting tokens recur in the training loss.
- Choice points in MTP loss: Explain what a choice point is and trace how predicting multiple future tokens makes its error recur in the loss, encouraging the model to handle consequential transitions better.
Why it matters: It connects MTP’s training objective to the section’s claim that the model can become better at forecasting important turns in a sequence.
Key terms: choice point, MTP loss, multi-token prediction, planning
5.2.4 Higher inference speed via speculative decoding
Explains how MTP heads could accelerate inference through speculative decoding, while distinguishing that possibility from how DeepSeek V3 was publicly released.
- Speculative decoding workflow: Explain how speculative decoding replaces repeated single-token generation with drafting several candidate tokens and verifying them together, and why MTP heads can supply the draft.
Why it matters: It connects MTP's multiple-token predictions to a possible inference-speed benefit. - DeepSeek V3 inference choice: Distinguish MTP's potential use for speculative decoding from DeepSeek V3's reported public-release inference setup.
Why it matters: It prevents a possible speedup from being mistaken for the inference method used in the public release.
Key terms: speculative decoding, draft model, MTP heads, verification, forward pass, pre-training, standard single-token prediction, MTP modules
5.3 The DeepSeek MTP architecture: A visual and mathematical walkthrough
This section contrasts independent multi-token output heads with DeepSeek’s sequential MTP modules and clarifies how the prediction chain is formed during training.
- Sequential MTP prediction chain: Explain how DeepSeek’s sequential MTP modules differ from independent output heads: each module builds on the previous module’s hidden state, so the representation used for token t+1 informs the prediction of t+2. Clarify that training uses ground-truth tokens through teacher forcing, not the model’s sampled predictions.
Why it matters: The distinction explains why DeepSeek’s future-token predictions can be more coherent without confusing teacher forcing with feeding back sampled outputs.
Key terms: MTP modules, independent output heads, hidden state, causal chain, teacher forcing, ground-truth tokens
5.3.1 The starting point: The shared transformer trunk
This section identifies the shared Transformer trunk’s output, h⁰, as the starting point for multi-token prediction.
- Shared trunk hidden states: Trace how the input sequence passes through the shared Transformer trunk to produce h⁰, and explain why those hidden states—not raw tokens—start the MTP process.
Why it matters: Identifying h⁰ makes clear what the MTP modules receive from the main model.
Key terms: Shared Transformer Trunk, hidden state, context vector, Hidden state 0, h⁰
5.3.2 The MTP modules: A sequential chain of prediction
This section traces how DeepSeek-V3 chains MTP modules, each combining a previous hidden state with a future-token embedding to produce a prediction and a refined state for the next module.
- Causal MTP module chain: Explain how DeepSeek-V3 uses one MTP module per prediction depth and passes each module's refined hidden state to the next rather than using independent prediction heads.
Why it matters: The chain explains what makes this implementation different from separate heads that each predict from the same starting state. - MTP head input construction: Trace how a head takes the previous depth's hidden state and, during training, the ground-truth future-token embedding, then normalizes, concatenates, and projects them into its transformer input.
Why it matters: This shows precisely how information from the preceding depth and the future token enters each module. - Refined state and token logits: Describe how the dedicated transformer block produces a new hidden state and how that state both feeds the next MTP module and produces future-token logits through the shared output head.
Why it matters: The two uses of the refined state explain how a head makes a prediction without breaking the chain of contextual information.
Key terms: MTP Modules, prediction depth, causal chain, refined hidden state, hᵏ⁻¹, ground-truth embedding, RMS Norm, Merged Embedding, Linear Projection Layer, Mₖ
5.3.3 The final loss calculation
This subsection explains how the losses for multiple future-token predictions are combined during training.
- Summed multi-token training loss: Explain how training calculates the loss for one input token across D prediction depths and why that gives the model a broader training signal.
Why it matters: The combined loss trains the model to predict beyond only the next token.
Key terms: D prediction depths, ground-truth tokens, cross-entropy losses, total loss
5.4 Implementing a causal multi-token prediction module from scratch
This section describes a self-contained PyTorch implementation of causal multi-token prediction, from its modules and shared layers to its forward pass, loss, and sequence-length check.
- MTP module and model organization: Describe how a dedicated MTP module fits into the full model: it refines a previous hidden state using a future-token embedding, while the model provides one module per prediction depth and shares its embedding and output layers.
Why it matters: This organization connects each future prediction step to the same main model without giving every step separate embedding and output layers. - Causal forward pass and loss: Trace how the trunk's hidden state passes through successive MTP modules to produce future-token logits, and describe how their losses join the main next-token loss. Explain what the shrinking logit sequence lengths check.
Why it matters: The forward pass and loss show how the chain supplies training predictions beyond the main next-token prediction.
Key terms: DeepSeekMTPModule, main Transformer trunk, mtp_modules, shared embedding and output layers, h_main, h_prev, h_curr, prediction depth, MTP losses, causal chain
5.5 Quantization: Trading precision for speed and memory
Introduces FP8 quantization as a low-precision trade-off that helps make DeepSeek’s large-scale computation affordable.
- Low-precision efficiency trade-off: Explain why the section presents FP8 quantization as a complement to MLA, MoE, and MTP: accepting lower numerical precision can improve speed and memory efficiency, helping make large-scale training affordable.
Why it matters: This explains why numerical precision belongs alongside the model’s architectural choices in a discussion of training cost.
Key terms: FP8 quantization, low precision, speed, memory efficiency
5.5.1 What is quantization?
This section defines quantization as representing model parameters with fewer bits and explains the resulting trade-off between precision, memory use, and performance.
- Quantization and precision trade-off: Explain how quantization converts model parameters to a lower bit-width, using FP32 to FP16 as an example, and why a model may still perform well despite losing some numerical precision.
Why it matters: This trade-off explains why storing model parameters in lower-precision formats can be useful.
Key terms: quantization, FP32, FP16, bit-width, precision
5.5.2 Why quantize? The memory cost of high-precision parameters
This section explains why quantization is useful: lower-precision parameters reduce model memory needs and can speed computation, at the cost of some precision.
- Quantization memory and speed tradeoff: Explain how reducing parameter precision saves memory, why that can speed training and inference and make large models more accessible, and what is traded away.
Why it matters: This tradeoff explains why lower-precision formats are worth considering for large models.
Key terms: quantization, precision, GPU's main memory
5.5.3 Understanding numerical formats: The building blocks of quantization
This section explains how numerical formats allocate bits to range and precision, and why those choices matter for quantization.
- Floating-point bit roles: Explain what the sign, exponent, and mantissa contribute to a floating-point number, and how exponent and mantissa bits affect range and precision.
Why it matters: The bit allocation explains the trade-offs among the formats in this section. - FP32 FP16 BF16 trade-offs: Distinguish FP32, FP16, and BF16 by their exponent and mantissa allocations, and explain the resulting range and precision trade-offs.
Why it matters: These comparisons show why using fewer bits does not always sacrifice range and precision in the same way. - INT8 and FP8 distinction: Distinguish INT8’s uniformly spaced integer values from FP8’s floating-point representation, and explain the limitation of INT8 when values span widely different magnitudes.
Why it matters: The distinction motivates why an eight-bit floating-point format can be useful where a plain eight-bit integer loses information.
Key terms: sign, exponent, mantissa, significand, dynamic range, precision, FP32, FP16, BF16, overflow
5.5.4 The basic mechanism: Scaling
This section explains how scaling maps FP32 values into a symmetric INT8 range and approximately recovers them afterward.
- Tensor scaling for INT8 quantization: Explain how to find a tensor’s maximum absolute value, use it to scale FP32 values into the −127 to 127 range, round them to integers, and approximately recover them by dividing by the same factor.
Why it matters: This scaling principle is the foundation for the fine-grained quantization discussed next.
Key terms: scaling, Maximum Absolute Value (α), scaling factor, quantize, de-quantize, FP32
5.6 The five pillars of DeepSeek's FP8 training
This section introduces DeepSeek’s FP8 training as a five-pillar framework and previews the explanation of each pillar.
5.6.1 Pillar 1: The mixed precision framework
This section explains how DeepSeek uses FP8 for fast linear-layer matrix multiplications while retaining higher precision where training is sensitive to numerical error.
- Mixed precision trade-off: Explain why DeepSeek assigns different numerical formats to different parts of training rather than using FP8 or FP32 everywhere.
Why it matters: The speed benefit depends on using low precision selectively, not indiscriminately. - Forward and input-gradient flow: Trace the precision choices for the linear layer’s forward pass and its backward input-gradient calculation.
Why it matters: The same fast-computation, safer-result pattern applies in both directions through the layer. - Weight gradients and updates: Distinguish the precision used to calculate the weight gradient from the precision used to store it and update the master weights.
Why it matters: The weight update changes the model’s lasting parameters, so errors there are especially consequential. - Quantization-sensitive module exceptions: Describe why DeepSeek exempts certain Transformer modules from FP8 quantization and identify the kinds of modules named.
Why it matters: The exceptions show that mixed precision is chosen by sensitivity, not simply applied to every module.
Key terms: mixed precision, FP8, FP32, BF16, GEMM, Fprop, Dgrad, activations, dL/dx, FP32 accumulator
5.6.2 Pillar 2: Fine-grained quantization
This section explains how DeepSeek uses separate scales for small groups of activations and blocks of weights to protect precision during low-precision training.
- Outlier sensitivity of tensor scaling: Explain why one scale for an entire tensor can erase distinctions among smaller values when the tensor contains an outlier.
Why it matters: This failure explains why DeepSeek uses more than one scaling factor. - Independent activation and weight scales: Describe how fine-grained quantization partitions activations and weights and why each partition receives its own scale.
Why it matters: Local scales preserve differences that a distant outlier could otherwise obscure. - Fine-grained computation and de-quantization: Trace the section’s workflow from separately scaled inputs and weights through low-precision matrix multiplication to de-quantized output.
Why it matters: It connects the local scaling strategy to the matrix multiplication and its usable output.
Key terms: tensor-wise scaling, outliers, scaling factor, quantize, Fine-Grained Quantization, activations, weights, groups, tiles, FP8
5.6.3 Pillar 3: Increasing accumulation precision
This section explains why long FP8 matrix multiplications can lose accuracy during accumulation and how DeepSeek combines Tensor Cores with FP32 accumulation on CUDA Cores to reduce that error.
- Low precision accumulation problem: Explain why accumulating a long FP8 dot product entirely in a Tensor Core can produce numerical error, even though the hardware performs FP8 multiplication quickly.
Why it matters: Fast FP8 multiplication alone does not guarantee an accurate matrix-multiplication result. - Periodic FP32 promotion: Trace how DeepSeek divides accumulation between Tensor Cores and CUDA Cores, including when partial sums are transferred and where the running total is kept.
Why it matters: Periodic promotion preserves much of the speed of Tensor Core computation while reducing accumulation error. - High precision output finishing: Describe how scaling factors are applied after high-precision accumulation and what form the resulting value takes for the next layer.
Why it matters: This connects FP8 computation and precise accumulation to the value passed onward through the network.
Key terms: FP8, dot product, Tensor Cores, accumulator, Low Prec Acc, promotion to CUDA Cores, WGMMA, Nc interval, FP32 Register, Scaling Factors
5.6.4 Pillar 4: Mantissa over exponents
This section explains why DeepSeek uses the more precise E4M3 format throughout FP8 training instead of switching formats for backward passes.
- FP8 format trade-off: Distinguish E5M2 from E4M3 and explain why a conventional mixed-precision approach used different formats for forward and backward passes.
Why it matters: The usual format choice makes clear what DeepSeek changed and why. - Uniform E4M3 through local scaling: Explain how fine-grained quantization lets DeepSeek use E4M3 in both forward and backward passes despite its smaller dynamic range.
Why it matters: It shows how local scaling makes a consistent, higher-precision FP8 format practical.
Key terms: E5M2, E4M3, exponent, mantissa, dynamic range, precision, fine-grained quantization, outliers, scaling factors, forward pass
5.6.5 Pillar 5: Online quantization
This section explains why DeepSeek calculates FP8 scaling factors from the current tensor rather than relying on historical values.
- Delayed quantization risks: Explain how delayed quantization chooses a scaling factor and why a changing data distribution can make that factor unsuitable for the current tensor.
Why it matters: The failure of stale scaling factors explains why DeepSeek calculates them from current data. - Online scaling workflow: Describe how online quantization derives and applies a scaling factor for the current tensor, and explain its computational trade-off.
Why it matters: A small amount of extra work lets the scale track the data being quantized, improving FP8 numerical stability.
Key terms: Delayed Quantization, scaling factor, historical maximum, overflow, underflow, Online Quantization, maximum absolute value, current tensor
5.7 Summary
Where this is used later
- Building the components: RoPE, MLA, MoE, and MTP (chapter 6) · The multi-token prediction architecture introduced earlier.
- The final architecture: MiniDeepSeek (chapter 6) · The sequential chain of multi-token-prediction modules.
- The cost of intelligence (chapter 8) · Introduces quantization, one of the alternative approaches named here.
- Configuration and system setup (chapter 6) · Introduces the idea of a mixed-precision training framework.
Concepts explained in this chapter
- FP32Float32, also called FP32, is a 32-bit floating-point numerical format that uses four bytes per value.
- mixed precisionMixed precision is a computation technique that uses multiple numerical precisions to reduce resource use while maintaining training effectiveness.
- FP8An 8-bit floating-point numerical format used to represent and compute neural-network values with reduced precision.
- multi-token predictionMulti-token prediction is a language-model training or inference technique that predicts multiple future tokens rather than only the next token.
- dynamic rangeDynamic range is the span between the smallest and largest magnitudes that a numerical representation or computation can represent effectively.
- dequantizationDequantization is the transformation of quantized values back into a higher-precision numerical representation using quantization scales.
- fine-grained quantizationFine-grained quantization reduces numerical precision independently within small groups of activations or blocks of weights, assigning each region its own scaling factor.
- bfloat16BF16 is a 16-bit floating-point numerical format with a wide exponent range commonly used for machine-learning computation.
- causation chainA causal chain is an ordered sequence of prediction stages in which each stage conditions on the hidden state produced by the preceding stage.
- stateA state is the information describing the current situation from which a reinforcement-learning policy selects an action.
- anomalyAn outlier is a numerical activation, weight, or gradient whose magnitude is unusually large relative to other values in its group.
- MTP moduleAn MTP module is a neural-network component that predicts an additional future token from a representation produced at the preceding prediction depth.
- precisionPrecision is the fineness with which a numerical format represents significant digits and distinguishes nearby values.
- E4M3E4M3 is an FP8 encoding with four exponent bits and three explicitly stored mantissa bits.
- backpropagationA backward pass is the computation that propagates gradients backward through a neural network to support parameter optimization.
- forward passA forward pass is the computation that propagates input data through a neural network to produce model outputs.
- master weightA master weight is a high-precision persistent copy of a neural-network parameter used as the source for optimization updates.
- matrix-matrix multiplicationMatrix multiplication is an algebraic operation that combines rows of one matrix with columns of another to produce a matrix.
- weight gradientA weight gradient is the derivative of a training loss with respect to a neural-network weight parameter.
- FP32 registerAn FP32 register is a hardware register capable of storing a 32-bit floating-point value.
- Tensor CoreA Tensor Core is a GPU computational unit specialized for high-throughput matrix multiplication and accumulation, typically at reduced numerical precision.
- WGMMAWGMMA is an NVIDIA GPU instruction that performs warp-group-level matrix multiply-accumulate operations.
- accumulatorAn accumulator is a hardware register that stores a running sum of products during a computation.
- gpu coreA CUDA core is a general-purpose arithmetic processing unit in an NVIDIA GPU that supports a broad range of computations.
- promotion to CUDA CoresPromotion to CUDA Cores is a mixed-precision accumulation technique that periodically transfers partial sums from low-precision Tensor Core computation to high-precision CUDA Core computation.
- quantizationQuantization is the process of representing higher-precision numerical values with lower-precision values, typically by scaling and rounding them.
- scalar quantizationStandard quantization maps an entire tensor to a lower-precision numerical range using one shared scaling factor.
- exponentAn exponent is the component of a floating-point representation that determines the scale and dynamic range of represented values.
- mantissaA mantissa, or significand, is the component of a floating-point representation that determines the significant digits and precision of a value.
- linear layerA linear layer is a neural-network layer that maps input representations to outputs through a learned affine transformation.
- merged embeddingA merged embedding is a vector formed by concatenating normalized hidden-state and token-embedding representations.
- transformer blockA transformer block is a repeated neural-network unit combining attention, a feed-forward network, normalization, and residual connections.
- un-embedding matrixAn un-embedding matrix is a learned output projection that maps hidden representations to scores over a model's vocabulary.
- online quantizationOnline quantization calculates quantization scaling factors from the current tensor or batch during processing rather than from historical statistics.
- context vectorA context vector is a numerical representation summarizing information relevant to a model's prediction at a particular sequence position.
- total lossTotal loss is an aggregate training error formed by summing the individual losses associated with multiple prediction targets.
- underflowUnderflow is the condition in which a numerical value is too small in magnitude for a format to represent its detail accurately.
- additional headAn output head is the final neural-network component that transforms hidden representations into task-specific output scores or predictions.
- context matrixA context matrix contains attention outputs for all sequence positions, with each row formed by weighting value vectors.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
