How deeply does Deep Learning with Python, Third Edition cover next-token prediction?

Explains the objective through generation examples and shifted labels, then stops short of deriving the training algorithm.

Next-token prediction

As explained in Deep Learning with Python, Third Edition

A language-model training objective that teaches the model to predict the next token in a sequence from previous tokens.

Next-token prediction trains a language model to continue a sequence by selecting the most probable token after the tokens already provided. This makes a pretrained model behave like an autocomplete system rather than a naturally conversational agent. section 16.3.1

Prompting changes the likely continuation

A prompt such as “How can I make brownies?” can continue as a forum-style exchange, while “The following brownie recipe is easy to make in just a few steps. You can start by” makes recipe instructions more likely. The model is extending the text pattern, not interpreting the prompt conversationally. section 16.3.1

The practical consequences are straightforward:

Expect plausible continuation, not automatically useful answers; usefulness depends strongly on the prompt’s surrounding context. section 16.3.1
Treat each label as the next token in the sequence, shifted one position relative to the input token IDs. section 16.3.2
During instruction fine-tuning, compute loss for response tokens while excluding the fixed prompt and padded positions. section 16.3.2

How Deep Learning with Python, Third Edition covers it

Cover of Deep Learning with Python, Third Edition

liveBook manning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Introduction to Generative AI, Second EditionNext token prediction is a language-model training objective that predicts the subsequent token from preceding context.
liveBook manning.com

Build a Text-to-Image Generator (from Scratch)Next-token prediction is a sequence-generation technique that predicts the next discrete token from preceding tokens and conditioning information.
liveBook manning.com

Build a Large Language Model (From Scratch)A language-model training objective in which the model predicts the next token or word from preceding context.
liveBook manning.com

Learn Generative AI with PyTorchThe language-modeling objective of predicting the following token from the current and previous tokens in a sequence.
liveBook manning.com

How Large Language Models WorkAn autoregressive generation strategy that predicts each successive word or token from the preceding context.
liveBook manning.com

Deep Learning with R, Third EditionNext-token prediction is the task of predicting the next token in a sequence given the preceding tokens.
liveBook manning.com

Transformers in ActionNext-token prediction is the task of predicting the token that follows a given preceding token sequence.
liveBook manning.com

Reinforcement Learning from Human FeedbackNext-token prediction is a language-modeling objective that trains a model to predict the token following a given context.

Compare how 9 books explain next-token prediction →

Related concepts

language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
fine-tuningused by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
data representationproduces · A data representation is a form or encoding in which information is structured for storage, processing, transmission, or interpretation.
reinforcement learning with human feedbackcontrasts with · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.