liveBook · Manning

Prompt masking

As explained in Reinforcement Learning from Human Feedback

Prompt masking is the exclusion of prompt-token positions from the training loss while retaining response-token positions.

Prompt masking excludes prompt-token positions from the autoregressive training loss while retaining response-token positions. Instruction fine-tuning therefore optimizes the model to produce useful answers rather than to reproduce user queries, while preserving the same loss function used during pretraining. section 4.3

Multiturn training commonly applies one of two masking policies:

  • Final-turn only: include only the final assistant response in the loss; mask earlier context, including earlier assistant turns. section 4.3
  • User turns only: mask every user turn while training on every assistant response, including intermediate replies. section 4.3
  • Unroll long conversations into shorter examples by predicting one assistant response while masking prior context and excluding future turns. section 4.3

Masked instruction sequence

For a sequence containing a user query followed by an assistant answer, mask the query positions and apply the loss only to the answer positions. In a multiturn example, final-turn masking trains only the last assistant reply, whereas user-turn masking trains every assistant reply. section 4.3

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Distinguishes single-turn and two multiturn policies, but defers broader implementation details beyond masking and sequence construction.
Examples
Uses masked user queries, assistant responses, final-turn conversations, and unrolled multiturn training examples.
What it assumes
Assumes autoregressive language-model training and distinguishes instruction-tuning data from pretraining documents.
What you can do after
Apply response-focused loss and choose a final-turn or user-turn masking policy for multiturn fine-tuning.
How it connects
Connects instruction fine-tuning, pretraining, autoregressive loss, multiturn masking, and post-training.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.