liveBook · Manning

Principled alignment

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Principled alignment shapes model behavior around abstract human values and ethical principles.

A helpful answer can still cause harm, mislead a user, or conflict with a policy. Principled alignment shapes model behavior around foundational values such as honesty, helpfulness, and harmlessness. The engineering challenge is turning those abstract values into behavior that can be trained and evaluated. section 1.2.1 section 8.0

Principled alignment is one of three interacting pillars. Policy alignment enforces specific rules and procedures; personality alignment shapes tone and communication style. A customer service assistant, for example, must balance being helpful with following company procedures and maintaining a consistent voice. section 1.2.1 section 8.1.4

A preference can combine several judgments

An annotator compares two financial summaries. One cites the source and uses a neutral tone; the other invents a geographic explanation and sounds more confident. Preferring the first rewards honesty, adherence to citation instructions, and professional style at once. The single preference does not reveal how much each dimension contributed. section 7.1.2

Reinforcement learning from human feedback (RLHF) can learn values indirectly from aggregated preferences, but its blended signal makes individual pillars hard to control. Demonstrations alone can teach a model to imitate helpful answers without reliably establishing a general preference for helpfulness. section 7.1.2 section 7.5.2

Constitutional AI makes values explicit in a set of principles. An AI critic uses them to critique and revise responses and choose between alternatives; those outputs become training data. For example, a directive to choose the more honest response places honesty directly in the judgment process. This is an operational use of the principle, not evidence that the critic philosophically understands honesty. section 8.0 section 8.1.4

Turn a value into a training signal

For a financial research assistant, write a principle not to invent facts absent from the source. An AI critic can use it to label variants of single-quarter questions, multi-quarter trends, and cross-company comparisons. Check the critic carefully: systematic errors in its judgments can be amplified by training. section 8.2.5

Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
Technique
Principled
Policy-based
Personality
RLHF
Indirect, through average labeler’s preferences
Implicit, through refusal demonstrations
Strong, through the demonstration style
RLAIF
Indirect, through the rubric the AI judge reads
Same as RLHF
Same as RLHF, plus AI judge bias
Constitutional AI
Direct, principles state values explicitly
Encodable as principles, with interpretive risk
Encodable, leaks the AI critic’s style priors

Compare how preference-based and principle-based methods act on the three alignment pillars, including their interpretive risks. section 8.1.4

Explicit principles do not eliminate judgment calls. A critic may interpret procedural rules inconsistently on new inputs, and its own style preferences can influence personality. Evaluate outcomes by pillar rather than treating an aggregate improvement as proof that every dimension improved. section 8.1.4 section 7.1.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains explicit principles, critic-generated training signals, and their limitations, contrasting them with preference-based alignment.
Examples
Uses financial-summary preferences and a source-grounding principle for a financial research assistant.
What you can do after
Evaluate alignment signals by pillar and identify risks in AI-critic judgments.
How it connects
Connects Constitutional AI and reinforcement learning from human feedback to policy and personality alignment.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.