liveBook · Manning

Constitutional AI

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An AI alignment technique that guides model behavior with a set of explicit principles and uses AI-generated feedback during training.

A model trained on human preferences may encounter harmful requests that labelers never saw, while collecting comparisons for every variant is expensive. Constitutional AI addresses this by using an AI critic guided by a written constitution: a set of plain-language principles such as honesty and harmlessness. The critic generates revisions and preferences that become training data. section 8.0 section 8.1.4

The method builds on reinforcement learning from human feedback (RLHF), changing the source of training signals rather than replacing the underlying optimization machinery. Humans still choose the principles; the AI applies them to individual responses. section 8.2 section 8.2.5

The full recipe has two training phases:

  • Supervised learning (SL-CAI): Generate responses to prompts likely to elicit harm. Critique and revise each response against randomly selected principles, repeating with different principles. Fine-tune the pretrained model on the resulting prompt–revised response pairs. section 8.2.1
  • Reinforcement learning (RL-CAI): Sample two responses per prompt from the SL-CAI model. Ask an AI feedback model to choose the better response under the constitution, then train a reward model on those preferences. Optimize the policy with PPO, using SL-CAI as the reference for a KL penalty that constrains changes. section 8.2.1
Figure 8.1 The Constitutional AI two-phase pipeline. Phase 1 generates critique-and-revise traces from a helpful-only baseline guided by the constitution, then fine-tunes the pretrained model on the <prompt, revised responses> pairs to produce the SL-CAI model. Phase 2 samples paired responses from the SL-CAI model, asks the model to pick the preferred response under the constitution, trains a reward model on those AI preferences, and runs PPO with a KL penalty against the SL-CAI reference. The downstream RL machinery is identical to InstructGPT.

Trace how critique-revised responses and AI preferences feed the two training phases. section 8.2.1

An honesty principle

A financial research assistant may fabricate plausible-sounding names. An honesty principle can direct a critic to assess a response for factual accuracy and revise it before the response is used as training data. The principle guides the judgment; it does not guarantee that the critic detects every fabrication. section 8.0 section 8.1.4 section 5.4.5

A practical first pass can stop after supervised fine-tuning on critique-revised responses. A stronger critic can generate revisions for a smaller model to learn from; the reinforcement-learning phase adds reward modeling, preference collection, and PPO complexity. section 8.5.2

Check the resulting model and critic for these failure modes:

  • Coverage gaps: Unwritten expectations remain unconstrained. Use red teaming to find failures and add relevant principles. section 8.3.3 section 8.5.5
  • Stylistic leakage: The critic’s preferences for tone or formatting can enter the labels. Sanity-check with another critic or add explicit style principles. section 8.3.3
  • Interpretation drift: Ambiguous principles may be applied inconsistently across prompts. Test them on adversarial prompts and rewrite them to reduce ambiguity. section 8.3.3

Calibrate the AI critic against human judgment: its biases can be amplified during optimization. Report helpfulness and harmlessness separately rather than assuming that a principle guarantees either. In the reported results, RL-CAI was more harmless than a helpful-only RLHF baseline and less evasive than a human-labeled harmlessness model. section 8.6 section 8.2.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Details Constitutional AI’s critique-revision and AI-preference phases, including reward modeling, PPO, practical adoption, and failure checks.
Examples
Uses a financial research assistant that fabricates names and an honesty principle directing the critic’s judgments.
What it assumes
Assumes familiarity with RLHF vocabulary, including reward models, preference learning, and KL constraints.
What you can do after
Apply Constitutional AI by designing a critique-revision fine-tuning pipeline and assessing whether to add AI-preference reinforcement learning.
How it connects
Connects Constitutional AI to RLHF, supervised fine-tuning, reward modeling, red teaming, and deliberative alignment.

Also explained in

Compare how 3 books explain Constitutional AI →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.