liveBook · Manning

Personality alignment

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Personality-based alignment shapes consistent traits and communication styles in AI behavior.

An assistant can follow a policy yet sound inconsistent or inappropriate for its role. Personality alignment shapes consistent traits and communication styles, including tone, contextual adaptation, and refusal style. section 1.5 section 7.1.2 section 8.1.4

Personality is one of three alignment dimensions. Principled alignment concerns values such as helpfulness and harmlessness; policy-based alignment concerns specific rules and procedures. The dimensions interact, so a friendly voice must still accommodate policy requirements and situations such as delivering bad news. section 1.2.1 section 7.1.2

Customer service voice

A customer service bot needs to maintain a brand voice while following company procedures and avoiding harm. When it delivers bad news, its friendly personality must work alongside those other obligations. section 1.2.1

Personality can be specified and learned in different ways. A behavioral specification can define a default persona, refusal style, and conditions for developer overrides. Supervised fine-tuning on labeler demonstrations can establish personality through the demonstrations’ style; the rubric and labelers’ interpretation of it shape what the model imitates. section 8.4.3 section 7.5.2

Preference-based training can improve several alignment dimensions at once, but a single preference choice blends them. An improvement in the aggregate does not reveal which dimension changed or whether a more helpful model also became more sycophantic. Labeler disagreement and unclear rubrics can further complicate diagnosis. section 7.1.2 section 7.5.2

Constitutional AI can express a personality requirement as a principle, such as using a neutral, professional tone. Its AI critic must still interpret that requirement, so the resulting behavior can inherit the critic’s stylistic preferences. section 8.1.4 section 8.2.5

Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
Technique
Principled
Policy-based
Personality
RLHF
Indirect, through average labeler’s preferences
Implicit, through refusal demonstrations
Strong, through the demonstration style
RLAIF
Indirect, through the rubric the AI judge reads
Same as RLHF
Same as RLHF, plus AI judge bias
Constitutional AI
Direct, principles state values explicitly
Encodable as principles, with interpretive risk
Encodable, leaks the AI critic’s style priors

Compare how preference-based and principle-based techniques affect personality alongside the other alignment pillars. section 8.1.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines the dimension and compares specification, demonstrations, and AI-critic interpretation without providing an implementation procedure.
Examples
Uses a customer service bot’s brand voice and a Model Spec’s default persona and refusal style.
What you can do after
Evaluate and understand how blended preference signals and critic style can obscure control over personality.
How it connects
Connects personality alignment to principled alignment, policy-based alignment, supervised fine-tuning, RLHF, and Constitutional AI.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.