A demonstration is a training example showing a model how to respond in a particular situation or case. Alignment fine-tuning uses demonstrations, including critique-revised examples, to teach desired behavior. section 8.5.3
Demonstrations can create shallow alignment when a model learns their surface patterns instead of the underlying principle. It may respond correctly to familiar prompts but generalize poorly to new settings. section 8.5.3
Use demonstrations diagnostically by checking:
- Whether the response matches a familiar training pattern or applies the intended principle to a held-out prompt. section 8.5.3
- Whether additional material explains the specification and its rationale before critique-revised demonstrations are used. section 8.5.3
Figure 8.4 Model spec midtraining inserted between pretraining and alignment fine-tuning. The pretrained model first trains on synthetic documents about the spec (articles, dialogues, examples). The midtrained model then proceeds through the standard alignment fine-tuning steps (SFT on critique-revised demonstrations, optionally RLAIF). The midtrained model generalizes the spec to held-out prompts in ways models without MSM do not.
Shows demonstrations positioned in the alignment pipeline after model-spec midtraining and before optional RLAIF. section 8.5.3








