Human preference labels cannot cover every failure an AI assistant may encounter. A constitution makes desired behavior explicit: it is a written list of natural-language principles specifying values, constraints, or behaviors that an AI critic can apply to responses. A more detailed specification can also turn individual rules into tests for a deployed model. section 8.1.3 section 8.4.4
A constitution can guide both training and evaluation:
- Write principles for the critic, such as “Do not invent factual claims that are not supported by the source” or “Acknowledge uncertainty.” section 8.1.3
- Generate supervised data: draw a principle, critique a response against it, revise the response, and fine-tune on the revisions. section 8.2.2 section 8.6
- Generate preference data: give an AI labeler a prompt, two candidate responses, and a randomly drawn principle; use its preference to train a reward model. section 8.2.4
- Turn specific clauses into evaluation cases and add discovered failures to a regression suite. Run the suite when the model changes. section 8.4.4
Critique and preference instructions serve different jobs: one asks what is wrong and how to revise it; the other asks which response better satisfies a principle. In the original critique-revision process, each response is assessed using a randomly drawn principle, not the entire constitution at once. section 8.2.2
Testing a financial research assistant
A specification might require a citation to the source paragraph for every numerical claim and an admission of uncertainty when the retrieved context lacks an answer. Test the first clause with earnings prompts and citation checks; test the second with questions whose answers are absent from the retrieved transcript. Add failures to the regression suite. section 8.4.4
Writing principles down makes them auditable, but it does not guarantee that the critic applies them as intended. Unspecified behaviors can remain uncontrolled; the critic’s preferred style can leak into labels; and ambiguous principles can be interpreted inconsistently. Add principles for observed gaps, check labels with a different critic, and test ambiguous rules on adversarial prompts before scaling. New principles can also interact in unexpected ways. section 8.3.1 section 8.3.3
Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
|
Technique
|
Principled
|
Policy-based
|
Personality
|
|
RLHF
|
Indirect, through average labeler’s preferences
|
Implicit, through refusal demonstrations
|
Strong, through the demonstration style
|
|
RLAIF
|
Indirect, through the rubric the AI judge reads
|
Same as RLHF
|
Same as RLHF, plus AI judge bias
|
|
Constitutional AI
|
Direct, principles state values explicitly
|
Encodable as principles, with interpretive risk
|
Encodable, leaks the AI critic’s style priors
|
Compare how written principles, human preferences, and AI feedback affect values, procedural policies, and personality. section 8.1.4

