liveBook · Manning

Next-word prediction

Also called next-token prediction

Next-token prediction is a language-modeling task or objective in which a model predicts the token that follows a given context.

How 13 Manning books explain it

Introduction to Generative AI, Second Edition

Next token prediction is a language-model training objective that predicts the subsequent token from preceding context.

How it is used
Explains the training objective behind text generation, pretraining, and supervised fine-tuning.
How deep it goes
Defines the objective and describes repeated guessing and weight adjustment without detailing how loss is calculated.
Examples
Contrasts honeymoon prompts with and without later context and describes a false-premise question about the Golden Gate Bridge.
What you can do after
Evaluate and understand how next-token training supports generation while leaving models vulnerable to confident hallucinations.

Read how Introduction to Generative AI, Second Edition explains next-word prediction →

Build a Text-to-Image Generator (from Scratch)

Next-token prediction is a sequence-generation technique that predicts the next discrete token from preceding tokens and conditioning information.

How it is used
Applies sequential generation to translation, image captioning, and transformer-based text-to-image generation.
How deep it goes
Explains shifted training targets, cross-entropy loss, causal masking, and autoregressive decoding.
Examples
Uses German-to-English translation, image captions such as “a dog runs,” and sequential image-patch generation.
What you can do after
Builds encoder–decoder transformers that train on shifted token targets and generate outputs one token at a time.

Read how Build a Text-to-Image Generator (from Scratch) explains next-word prediction →

Build a Large Language Model (From Scratch)

A language-model training objective in which the model predicts the next token or word from preceding context.

How it is used
Trains GPT-like language models on massive unlabeled text before downstream fine-tuning.
How deep it goes
Explains self-labeling, decoder-only generation, shifted tensors, sliding windows, batching, and masking without deriving optimization mathematics.
Examples
Demonstrates shifted token IDs, decoded word contexts, sliding windows, PyTorch tensors, batches, and stride-controlled overlap.
What you can do after
Build it from scratch by generating tokenized input-target pairs and batched PyTorch data loaders for GPT training.

Read how Build a Large Language Model (From Scratch) explains next-word prediction →

Learn Generative AI with PyTorch

The language-modeling objective of predicting the following token from the current and previous tokens in a sequence.

How it is used
Trains sequence models to predict the following token from the current and preceding tokens.
How deep it goes
Shows how to form shifted training pairs but does not specify a loss formula or its calculation.
Examples
Uses “how are you” with target “are you today,” plus LSTM, translation, and GPT training examples.
What you can do after
Build it from scratch by creating token sequences and one-step-shifted targets for training a text generator.

Read how Learn Generative AI with PyTorch explains next-word prediction →

Machines that Think

Next-token prediction is the task of predicting the token that follows a given sequence of preceding tokens.

How it is used
Frames language generation and diverse tasks as prediction of continuations from preceding context.
How deep it goes
Explains the objective and its reliability implications without specifying a loss formula or implementation.
Examples
Illustrates pattern extension with animal classification and examines mathematical skill development using MathCAMPS.
What you can do after
Evaluate and understand why fluent continuations can be useful without guaranteeing factual correctness.

Read how Machines that Think explains next-word prediction →

Understanding AI

Next-token prediction is the generation or selection of the most likely token to follow a preceding token sequence.

How it is used
Frames AI responses as context-based continuations, helping readers guide outputs and question unreliable ones.
How deep it goes
Explains the iterative generation loop and pretraining error-adjustment process without detailing network internals.
Examples
Uses story openings, code prefixes, formal letters, and a crowd that “couldn’t believe” a performance.
What you can do after
Evaluate/understand AI outputs as likely continuations rather than automatic guarantees of correctness.

Read how Understanding AI explains next-word prediction →

How Large Language Models Work

An autoregressive generation strategy that predicts each successive word or token from the preceding context.

How it is used
Uses sequential prediction to generate language, support information lookup, and provide brainstorming while exposing factual risks.
How deep it goes
Explains context-based association and training use, but stops short of mathematical or architectural internals.
Examples
Uses context-elimination figures, an absurd spaghetti explanation, and Python and Modula-3 programs calculating pi.
What you can do after
Evaluate generated responses more cautiously, verify factual claims, and avoid relying on predictions for novel tasks.

Read how How Large Language Models Work explains next-word prediction →

Deep Learning with R, Third Edition

Next-token prediction is the task of predicting the next token in a sequence given the preceding tokens.

How it is used
Trains language models to continue sequences by predicting the next token from preceding tokens.
How deep it goes
Explains the objective through text continuation and prompting, but does not detail loss computation or optimization.
Examples
Uses a brownie-recipe continuation and a novel-puzzle example to illustrate generation and familiarity limits.
What you can do after
Understand why plausible text continuation does not guarantee useful answers or success on unfamiliar problems.

Read how Deep Learning with R, Third Edition explains next-word prediction →

Deep Learning with Python, Third Edition

A language-model training objective that teaches the model to predict the next token in a sequence from previous tokens.

How it is used
Uses next-token prediction as the objective behind pretrained text generation and response-only instruction fine-tuning.
How deep it goes
Details next-token labels and response-only loss masking at the preprocessing level.
Examples
Uses brownie-recipe continuations and shifted token arrays to illustrate likely text and next-token labels.
What you can do after
Understand why a model trained to predict likely continuations may autocomplete a prompt rather than answer it conversationally.

Read how Deep Learning with Python, Third Edition explains next-word prediction →

Transformers in Action

Next-token prediction is the task of predicting the token that follows a given preceding token sequence.

How it is used
Trains decoder-only models to predict each token from the preceding sequence.
How deep it goes
Defines the prediction objective and connects it to causal attention and autoregressive generation, without specifying a loss calculation.
Examples
Names GPT as a decoder-only system that uses next-token prediction.
What you can do after
Understand how decoder-only models condition predictions on prior tokens and generate sequences autoregressively.

Read how Transformers in Action explains next-word prediction →

Applied Reinforcement Learning

A language-model training approach that predicts the next token from the preceding sequence.

How it is used
Frames the gap between fluent text generation and responses optimized for safety, helpfulness, or appropriateness.
How deep it goes
Defines the objective and traces token-by-token response training, but does not detail the prediction model's internals.
Examples
Uses contrasting mental-health replies and a medical-chatbot response ending with <EOS>.
What you can do after
Evaluate/understand why fluent next-token training alone does not encode preferences for safer, more helpful answers.

Read how Applied Reinforcement Learning explains next-word prediction →

Timeless Algorithms, The Foundational Ideas

A language-modeling objective in which a model learns to predict the next token from preceding tokens in a sequence.

How it is used
Uses next-token training as a foundation for transferring learned patterns to varied language tasks.
How deep it goes
Connects prediction to attention and autoregressive generation, but does not derive the loss mathematically.
Examples
Uses pronoun resolution, semantic association, factual recall, and chained generation to illustrate prediction.
What you can do after
Understand how a continuation objective can support varied tasks and why plausible predictions may still be unreliable.

Read how Timeless Algorithms, The Foundational Ideas explains next-word prediction →

Reinforcement Learning from Human Feedback

Next-token prediction is a language-modeling objective that trains a model to predict the token following a given context.

How it is used
Frames pretraining as capability acquisition and post-training as converting token continuation into useful conversation.
How deep it goes
Defines the objective conceptually and contrasts its behavior with question-answering, but does not derive its loss or implementation.
Examples
Uses OLMoE Instruct’s two versions and benchmark averages of 35 and 48 to illustrate post-training effects.
What you can do after
Evaluate/understand how pretraining and post-training divide capability acquisition from conversational behavior.

Read how Reinforcement Learning from Human Feedback explains next-word prediction →

Also covered in

What to know first

Related concepts

Questions about next-word prediction

What is next-word prediction?

Next-token prediction is a language-modeling task or objective in which a model predicts the token that follows a given context.

Which Manning books explain next-word prediction?

Introduction to Generative AI, Second Edition, Build a Text-to-Image Generator (from Scratch), Build a Large Language Model (From Scratch), Learn Generative AI with PyTorch, Machines that Think, Understanding AI, How Large Language Models Work, Deep Learning with R, Third Edition and 5 more.

What should I know before learning next-word prediction?

Manning books that teach next-word prediction usually introduce language model, transformer architecture, token and training first.

What is the difference between next-word prediction and multi-token prediction?

Next-token prediction is a language-modeling task or objective in which a model predicts the token that follows a given context. Multi-token prediction is a language-model training or inference technique that predicts multiple future tokens rather than only the next token.

What is next-word prediction also called?

Next-word prediction is also called next-token prediction.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.