liveBook · Manning

Deliberative alignment

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Deliberative alignment is an alignment approach that teaches a model to reason explicitly over principles and specifications during inference.

Deliberative alignment teaches a model the text of a specification and trains it to reason over relevant principles before answering. Unlike Constitutional AI, which embeds principles in model weights during training and omits them at inference, deliberative alignment carries principle-based reasoning into inference. section 8.5.5 section 8.6

Use deliberative alignment to distinguish where alignment work acts and what each method contributes:

  • Red teaming finds failures and unwritten expectations that can become new specification clauses; deliberative alignment then trains the model to follow those clauses. section 8.5.5
  • RLHF and RLAIF optimize training-time preferences, whereas deliberative alignment targets safer behavior through structured reasoning during generation. section 7.6.4
  • The approach depends on an explicit spec whose principles can guide reasoning; it does not eliminate the need to discover missing clauses or define desired behavior. section 8.5.5
Figure 8.6 Where each alignment technique acts in the model lifecycle. Model spec midtraining acts between pretraining and post-training, on documents about the spec. RLHF, Constitutional AI / RLAIF, and deliberative alignment are done during post-training time, all split into the SFT and RL phases. Deliberative alignment enables the model to reason over principles while generating.

Lifecycle placement clarifies how deliberative alignment differs from midtraining, RLHF, and Constitutional AI, and shows its role in enabling principle reasoning during generation. section 8.5.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines the training-versus-inference distinction and pipeline role, but does not detail the training algorithm or optimization procedure.
Examples
Uses specification clauses, red-team failure modes, and principle-based reasoning before answering as its concrete alignment artifacts.
What it assumes
Assumes familiarity with alignment methods and specification-based behavior, while contrasting deliberative alignment with RLHF and Constitutional AI.
What you can do after
Evaluate alignment methods by whether they encode principles in weights or invoke specification reasoning during generation.
How it connects
Connects specification design, red teaming, Constitutional AI, RLHF, RLAIF, model spec midtraining, and chain-of-thought reasoning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.