liveBook · Manning

Constitution-as-code document

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A constitution is an explicit set of natural-language principles that specifies the values or behavioral constraints an AI system should follow.

Human preference labels cannot cover every failure an AI assistant may encounter. A constitution makes desired behavior explicit: it is a written list of natural-language principles specifying values, constraints, or behaviors that an AI critic can apply to responses. A more detailed specification can also turn individual rules into tests for a deployed model. section 8.1.3 section 8.4.4

A constitution can guide both training and evaluation:

  • Write principles for the critic, such as “Do not invent factual claims that are not supported by the source” or “Acknowledge uncertainty.” section 8.1.3
  • Generate supervised data: draw a principle, critique a response against it, revise the response, and fine-tune on the revisions. section 8.2.2 section 8.6
  • Generate preference data: give an AI labeler a prompt, two candidate responses, and a randomly drawn principle; use its preference to train a reward model. section 8.2.4
  • Turn specific clauses into evaluation cases and add discovered failures to a regression suite. Run the suite when the model changes. section 8.4.4

Critique and preference instructions serve different jobs: one asks what is wrong and how to revise it; the other asks which response better satisfies a principle. In the original critique-revision process, each response is assessed using a randomly drawn principle, not the entire constitution at once. section 8.2.2

Testing a financial research assistant

A specification might require a citation to the source paragraph for every numerical claim and an admission of uncertainty when the retrieved context lacks an answer. Test the first clause with earnings prompts and citation checks; test the second with questions whose answers are absent from the retrieved transcript. Add failures to the regression suite. section 8.4.4

Writing principles down makes them auditable, but it does not guarantee that the critic applies them as intended. Unspecified behaviors can remain uncontrolled; the critic’s preferred style can leak into labels; and ambiguous principles can be interpreted inconsistently. Add principles for observed gaps, check labels with a different critic, and test ambiguous rules on adversarial prompts before scaling. New principles can also interact in unexpected ways. section 8.3.1 section 8.3.3

Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
Technique
Principled
Policy-based
Personality
RLHF
Indirect, through average labeler’s preferences
Implicit, through refusal demonstrations
Strong, through the demonstration style
RLAIF
Indirect, through the rubric the AI judge reads
Same as RLHF
Same as RLHF, plus AI judge bias
Constitutional AI
Direct, principles state values explicitly
Encodable as principles, with interpretive risk
Encodable, leaks the AI critic’s style priors

Compare how written principles, human preferences, and AI feedback affect values, procedural policies, and personality. section 8.1.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains principle drafting, random selection, preference labeling, clause-level testing, and failure mitigations.
Examples
Uses harmlessness critique prompts and a financial assistant specification requiring citations and admissions of uncertainty.
What you can do after
Evaluate and understand how written principles guide training and become behavioral tests, while recognizing critic bias and coverage gaps.
How it connects
Connects Constitutional AI to supervised fine-tuning, reinforcement learning from AI feedback, reward modeling, and regression testing.

Also explained in

Compare how 2 books explain constitution-as-code document →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.