liveBook · Manning

Next-word prediction

As explained in Applied Reinforcement Learning

A language-model training approach that predicts the next token from the preceding sequence.

Next-token prediction trains a language model to predict the next token from the preceding sequence. It produces fluent, coherent text by learning statistical patterns, but does not explicitly optimize for safety, helpfulness, or appropriateness. section 10.1

Fluent is not always better

For a mental-health query, “Consider talking to a professional” is plausible, but a response that acknowledges distress and gives clear emergency guidance may be preferable. Predicting likely text does not itself teach that preference. section 10.1

In supervised fine-tuning, the model still predicts response tokens one by one, conditioned on the prompt and earlier response tokens. The loss ignores prompt and padding tokens; an <EOS> target teaches the model where to stop. section 10.5

Figure 10.4 Supervised Fine-Tuning with label masking.

Label masking distinguishes prompt inputs from response tokens used for training. section 10.5

How Applied Reinforcement Learning covers it

How deep it goes
Defines the objective and traces token-by-token response training, but does not detail the prediction model's internals.
Examples
Uses contrasting mental-health replies and a medical-chatbot response ending with <EOS>.
What you can do after
Evaluate/understand why fluent next-token training alone does not encode preferences for safer, more helpful answers.
How it connects
Connects next-token prediction to language-model training, supervised fine-tuning, and reinforcement learning from human feedback.

Also explained in

Compare how 13 books explain next-word prediction →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.