A model trained on human preferences may encounter harmful requests that labelers never saw, while collecting comparisons for every variant is expensive. Constitutional AI addresses this by using an AI critic guided by a written constitution: a set of plain-language principles such as honesty and harmlessness. The critic generates revisions and preferences that become training data. section 8.0 section 8.1.4
The method builds on reinforcement learning from human feedback (RLHF), changing the source of training signals rather than replacing the underlying optimization machinery. Humans still choose the principles; the AI applies them to individual responses. section 8.2 section 8.2.5
The full recipe has two training phases:
- Supervised learning (SL-CAI): Generate responses to prompts likely to elicit harm. Critique and revise each response against randomly selected principles, repeating with different principles. Fine-tune the pretrained model on the resulting prompt–revised response pairs. section 8.2.1
- Reinforcement learning (RL-CAI): Sample two responses per prompt from the SL-CAI model. Ask an AI feedback model to choose the better response under the constitution, then train a reward model on those preferences. Optimize the policy with PPO, using SL-CAI as the reference for a KL penalty that constrains changes. section 8.2.1
Figure 8.1 The Constitutional AI two-phase pipeline. Phase 1 generates critique-and-revise traces from a helpful-only baseline guided by the constitution, then fine-tunes the pretrained model on the <prompt, revised responses> pairs to produce the SL-CAI model. Phase 2 samples paired responses from the SL-CAI model, asks the model to pick the preferred response under the constitution, trains a reward model on those AI preferences, and runs PPO with a KL penalty against the SL-CAI reference. The downstream RL machinery is identical to InstructGPT.
Trace how critique-revised responses and AI preferences feed the two training phases. section 8.2.1
An honesty principle
A financial research assistant may fabricate plausible-sounding names. An honesty principle can direct a critic to assess a response for factual accuracy and revise it before the response is used as training data. The principle guides the judgment; it does not guarantee that the critic detects every fabrication. section 8.0 section 8.1.4 section 5.4.5
A practical first pass can stop after supervised fine-tuning on critique-revised responses. A stronger critic can generate revisions for a smaller model to learn from; the reinforcement-learning phase adds reward modeling, preference collection, and PPO complexity. section 8.5.2
Check the resulting model and critic for these failure modes:
- Coverage gaps: Unwritten expectations remain unconstrained. Use red teaming to find failures and add relevant principles. section 8.3.3 section 8.5.5
- Stylistic leakage: The critic’s preferences for tone or formatting can enter the labels. Sanity-check with another critic or add explicit style principles. section 8.3.3
- Interpretation drift: Ambiguous principles may be applied inconsistently across prompts. Test them on adversarial prompts and rewrite them to reduce ambiguity. section 8.3.3
Calibrate the AI critic against human judgment: its biases can be amplified during optimization. Report helpfulness and harmlessness separately rather than assuming that a principle guarantees either. In the reported results, RL-CAI was more harmless than a helpful-only RLHF baseline and less evasive than a human-labeled harmlessness model. section 8.6 section 8.2.5


