Next-token prediction trains a model to predict the token following a sequence of preceding tokens. Combined with causal attention, it ensures that each prediction uses only available context. During generation, the predicted token returns to the input sequence, creating an autoregressive loop that can continue within the model’s context window. section 3.2
A decoder-only implementation connects the objective to several mechanisms:
- Causal attention masks future positions so each prediction depends only on preceding tokens. section 3.2
- Autoregressive generation feeds each newly produced token back into the model as part of the next input. section 3.2
- Tokenization converts raw strings into vocabulary IDs; byte pair encoding merges frequent character sequences into subword units. section 3.2
- Transformer blocks combine masked self-attention, normalization, multilayer processing, and residual connections before producing the next-token distribution. section 3.2
From text to generated sequence
An input encoder applies Unicode-safe byte encoding, BPE segmentation, and vocabulary lookup to produce token IDs. A decoder-only model processes those IDs with masked self-attention, predicts the next token, and repeats the process as generated tokens extend the sequence. section 3.2
Figure 3.1 Abstracted decoder-only architecture
Trace the recursive flow from preceding tokens through causal decoder processing to the next generated token. section 3.2
