Next-token prediction
Next-token prediction is the task or training objective of estimating the token that follows a sequence, given its preceding tokens and any conditioning information.
Introduction to Generative AI, Second Edition
Next token prediction is a language-model training objective that predicts the subsequent token from preceding context.
- How it is used
- Provides the training objective for generating text and remains the objective during supervised fine-tuning.
- How deep it goes
- Explains repeated guesses and weight adjustments, then contrasts autoregressive and bidirectional context without giving implementation details.
- Examples
- Uses honeymoon completions, an Eiffel Tower clue, and a false-premise Golden Gate Bridge question.
- What you can do after
- Evaluate/understand why fluent generated answers can still be wrong or fail to challenge a false premise.
Read how Introduction to Generative AI, Second Edition explains next-token prediction →
Build a Text-to-Image Generator (from Scratch)
Next-token prediction is a sequence-generation technique that predicts the next discrete token from preceding tokens and conditioning information.
- How it is used
- Applies sequential generation to translation, image captioning, and transformer-based text-to-image generation.
- How deep it goes
- Explains shifted training targets, cross-entropy loss, causal masking, and autoregressive decoding.
- Examples
- Uses German-to-English translation, image captions such as “a dog runs,” and sequential image-patch generation.
- What you can do after
- Builds encoder–decoder transformers that train on shifted token targets and generate outputs one token at a time.
Read how Build a Text-to-Image Generator (from Scratch) explains next-token prediction →
Build a Large Language Model (From Scratch)
A language-model training objective in which the model predicts the next token or word from preceding context.
- How it is used
- Trains GPT-like language models on massive unlabeled text before downstream fine-tuning.
- How deep it goes
- Explains self-labeling, decoder-only generation, shifted tensors, sliding windows, batching, and masking without deriving optimization mathematics.
- Examples
- Demonstrates shifted token IDs, decoded word contexts, sliding windows, PyTorch tensors, batches, and stride-controlled overlap.
- What you can do after
- Build it from scratch by generating tokenized input-target pairs and batched PyTorch data loaders for GPT training.
Read how Build a Large Language Model (From Scratch) explains next-token prediction →
Learn Generative AI with PyTorch
The language-modeling objective of predicting the following token from the current and previous tokens in a sequence.
- How it is used
- Trains recurrent networks and Transformers to model language and generate text one token at a time.
- How deep it goes
- Explains shifted targets, fixed windows, causal masking, and iterative generation, without deriving the loss mathematically.
- Examples
- Uses shifted text windows, the "how are you" example, PyTorch tensors, English-to-French translation, and GPT-2 training sequences.
- What you can do after
- Build it from scratch by preparing shifted token pairs, training a model, and generating text from a prompt.
Read how Learn Generative AI with PyTorch explains next-token prediction →
How Large Language Models Work
An autoregressive generation strategy that predicts each successive word or token from the preceding context.
- How it is used
- Uses sequential prediction to generate language, support information lookup, and provide brainstorming while exposing factual risks.
- How deep it goes
- Explains context-based association and training use, but stops short of mathematical or architectural internals.
- Examples
- Uses context-elimination figures, an absurd spaghetti explanation, and Python and Modula-3 programs calculating pi.
- What you can do after
- Evaluate generated responses more cautiously, verify factual claims, and avoid relying on predictions for novel tasks.
Read how How Large Language Models Work explains next-token prediction →
Deep Learning with R, Third Edition
Next-token prediction is the task of predicting the next token in a sequence given the preceding tokens.
- How it is used
- Explains the pretraining objective behind causal language models and their text-generation behavior.
- How deep it goes
- Defines next-token prediction and explains prompting consequences, but does not derive the training loss or token-level internals.
- Examples
- Uses advice, brownie-recipe, false-president, memorized-trick-question, and novel-puzzle examples to expose continuation and generalization behavior.
- What you can do after
- Evaluate/understand why generated text can be fluent, prompt-sensitive, plausible, and unreliable on unfamiliar problems.
Read how Deep Learning with R, Third Edition explains next-token prediction →
Deep Learning with Python, Third Edition
A language-model training objective that teaches the model to predict the next token in a sequence from previous tokens.
- How it is used
- Frames pretrained text generation as continuation prediction and motivates instruction fine-tuning for more useful responses.
- How deep it goes
- Explains the objective through generation examples and shifted labels, then stops short of deriving the training algorithm.
- Examples
- Uses Gemma continuations for advice, brownie recipes, and factual prompts, plus token-ID batches showing shifted next-token labels.
- What you can do after
- Evaluate generated text as probable continuation and inspect or prepare next-token targets during Keras-based fine-tuning.
Read how Deep Learning with Python, Third Edition explains next-token prediction →
Transformers in Action
Next-token prediction is the task of predicting the token that follows a given preceding token sequence.
- How it is used
- Supports open-ended generation and instruction-following systems built with decoder-only language models.
- How deep it goes
- Explains the objective, causal constraint, recursive generation loop, tokenization, and supporting implementation components.
- Examples
- Uses GPT-2-style encoding, BPE segmentation, masked attention code, transformer-block code, and recursive sequence generation.
- What you can do after
- Apply decoder-only generation by encoding text, masking future positions, predicting one token, and feeding it back repeatedly.
Read how Transformers in Action explains next-token prediction →
Reinforcement Learning from Human Feedback
Next-token prediction is a language-modeling objective that trains a model to predict the token following a given context.
- How it is used
- Frames pretraining as capability acquisition and post-training as converting token continuation into useful conversation.
- How deep it goes
- Defines the objective conceptually and contrasts its behavior with question-answering, but does not derive its loss or implementation.
- Examples
- Uses OLMoE Instruct’s two versions and benchmark averages of 35 and 48 to illustrate post-training effects.
- What you can do after
- Evaluate/understand how pretraining and post-training divide capability acquisition from conversational behavior.
Read how Reinforcement Learning from Human Feedback explains next-token prediction →
Related concepts
- language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- fine-tuningused by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
- pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
- data representationproduces · A data representation is a form or encoding in which information is structured for storage, processing, transmission, or interpretation.
- reinforcement learning with human feedbackcontrasts with · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.