How deeply does Reinforcement Learning from Human Feedback cover next-token prediction?

Defines the objective conceptually and contrasts its behavior with question-answering, but does not derive its loss or implementation.

Next-token prediction

As explained in Reinforcement Learning from Human Feedback

Next-token prediction is a language-modeling objective that trains a model to predict the token following a given context.

Next-token prediction trains a language model to produce the token most likely to follow a given context. Internet-scale pretraining uses this objective to build broad knowledge and capabilities, but the resulting model naturally continues text rather than conducting useful question-and-answer conversations. Post-training therefore reshapes the model’s behavior from token continuation toward conversational responses. section 1.4

From continuation to conversation

A base model may possess substantial knowledge while still responding through ordinary text continuation. Instruction tuning and later post-training make that capability accessible through question-answering formats; OLMoE Instruct’s post-training update improved its benchmark average from 35 to 48 without changing most pretraining. section 1.4

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Introduction to Generative AI, Second EditionNext token prediction is a language-model training objective that predicts the subsequent token from preceding context.
liveBook manning.com

Build a Text-to-Image Generator (from Scratch)Next-token prediction is a sequence-generation technique that predicts the next discrete token from preceding tokens and conditioning information.
liveBook manning.com

Build a Large Language Model (From Scratch)A language-model training objective in which the model predicts the next token or word from preceding context.
liveBook manning.com

Learn Generative AI with PyTorchThe language-modeling objective of predicting the following token from the current and previous tokens in a sequence.
liveBook manning.com

How Large Language Models WorkAn autoregressive generation strategy that predicts each successive word or token from the preceding context.
liveBook manning.com

Deep Learning with R, Third EditionNext-token prediction is the task of predicting the next token in a sequence given the preceding tokens.
liveBook manning.com

Deep Learning with Python, Third EditionA language-model training objective that teaches the model to predict the next token in a sequence from previous tokens.
liveBook manning.com

Transformers in ActionNext-token prediction is the task of predicting the token that follows a given preceding token sequence.

Compare how 9 books explain next-token prediction →

Related concepts

language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
fine-tuningused by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
pretrainingused by · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
data representationproduces · A data representation is a form or encoding in which information is structured for storage, processing, transmission, or interpretation.
reinforcement learning with human feedbackcontrasts with · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.