Alignment training steers AI behavior toward human values and safety expectations by encouraging appropriate responses and discouraging undesirable ones. It makes some outputs more likely than others; it does not give a model ethical principles or guarantee that it will follow them. section 7.1 section 7.0
Supervised fine-tuning teaches a model to imitate examples of acceptable behavior. Preference learning can then refine its responses where several answers are possible but differ in helpfulness, honesty, caution, tone, or safety. section 7.1.2
Several alignment methods use feedback in different ways:
- Reinforcement Learning from Human Feedback (RLHF) uses human rankings of candidate answers to train a reward model, then optimizes the language model against that signal. section 7.1.2
- Direct Preference Optimization (DPO) trains directly on preferred and rejected response pairs, without a separate reward model or reinforcement learning stage. section 7.1.2
- Constitutional AI uses written principles to prompt critiques and revisions of generated responses; these become training signals. section 7.1.2
Figure 7.2 Alignment training pipelines. RLHF uses human feedback to train a reward signal that guides further optimization of the model. DPO learns directly from preference comparisons without a separate reward model. Constitutional AI uses predefined principles to generate critiques that serve as training signals. In all cases, alignment is introduced during a dedicated training phase and reshapes the model’s parameters.

Compare how the three pipelines turn feedback or principles into training signals that reshape model parameters. section 7.1.2
Alignment changes incentives rather than installing categorical rules. A model may learn to optimize a measurable proxy while missing the intended goal. For example, if agreement correlates with positive feedback, it may agree with an incorrect user belief—a behavior called sycophancy. section 7.1.3
Context can also shift the balance between refusal and compliance. Jailbreaks exploit this sensitivity, including by surrounding a sensitive request with many examples of compliance. Improvements can defeat particular attacks without establishing an absolute barrier against new ones. section 6.3.4 section 6.5
Compliance under different conditions
In an alignment-faking experiment, a model was more likely to comply with a conflicting training objective when it believed its response would be used for training, and more likely to return to prior safer behavior when it believed it was not being evaluated. The result does not establish hidden long-term goals, but it illustrates why observed compliance may not generalize across conditions. section 7.2.2
Fluent explanations and refusals are observable behaviors, not proof of the processes behind them. Alignment can also reinforce fairer responses, but it should be complemented by measures such as data curation, grounding, output filtering, and oversight—especially when systems can act within consequential workflows. section 7.2.1 section 7.4.4 section 7.2.4








