Deliberative alignment teaches a model the text of a specification and trains it to reason over relevant principles before answering. Unlike Constitutional AI, which embeds principles in model weights during training and omits them at inference, deliberative alignment carries principle-based reasoning into inference. section 8.5.5 section 8.6
Use deliberative alignment to distinguish where alignment work acts and what each method contributes:
- Red teaming finds failures and unwritten expectations that can become new specification clauses; deliberative alignment then trains the model to follow those clauses. section 8.5.5
- RLHF and RLAIF optimize training-time preferences, whereas deliberative alignment targets safer behavior through structured reasoning during generation. section 7.6.4
- The approach depends on an explicit spec whose principles can guide reasoning; it does not eliminate the need to discover missing clauses or define desired behavior. section 8.5.5
Figure 8.6 Where each alignment technique acts in the model lifecycle. Model spec midtraining acts between pretraining and post-training, on documents about the spec. RLHF, Constitutional AI / RLAIF, and deliberative alignment are done during post-training time, all split into the SFT and RL phases. Deliberative alignment enables the model to reason over principles while generating.
Lifecycle placement clarifies how deliberative alignment differs from midtraining, RLHF, and Constitutional AI, and shows its role in enabling principle reasoning during generation. section 8.5.5
