A learning signal is numerical or evaluative feedback that guides model optimization. Scalar evaluation scores—such as faithfulness, answer relevance, or context precision—are structurally similar to signals consumed by reinforcement-learning algorithms when updating a policy. section 7.0
Common learning signals differ in where their feedback comes from:
- Evaluation metrics provide scalar scores that can become inputs to training. section 7.0
- Human preference comparisons train systems without requiring people to demonstrate the correct answer from scratch, as in RLHF. section 7.0
- Written principles become signals when an AI critic evaluates outputs against them, as in Constitutional AI. section 8.1.4
- AI-generated labels can train a reward model in a repeated loop of candidate generation, critique, policy improvement, and auditing. section 8.2.5
Financial research assistant
To reduce invented numbers in earnings summaries, a team writes a principle forbidding facts absent from the source. An AI critic labels variants covering single-quarter questions, multi-quarter trends, and cross-company comparisons; those labels supply training feedback without collecting human rankings for every example. section 8.2.5
Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
|
Technique
|
Principled
|
Policy-based
|
Personality
|
|
RLHF
|
Indirect, through average labeler’s preferences
|
Implicit, through refusal demonstrations
|
Strong, through the demonstration style
|
|
RLAIF
|
Indirect, through the rubric the AI judge reads
|
Same as RLHF
|
Same as RLHF, plus AI judge bias
|
|
Constitutional AI
|
Direct, principles state values explicitly
|
Encodable as principles, with interpretive risk
|
Encodable, leaks the AI critic’s style priors
|
Compare how RLHF, RLAIF, and Constitutional AI turn preferences or principles into signals affecting principled behavior, policy behavior, and personality. section 8.1.4



