liveBook · Manning

Learning signal

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A training signal is numerical or evaluative feedback that guides the optimization of a machine-learning model.

A learning signal is numerical or evaluative feedback that guides model optimization. Scalar evaluation scores—such as faithfulness, answer relevance, or context precision—are structurally similar to signals consumed by reinforcement-learning algorithms when updating a policy. section 7.0

Common learning signals differ in where their feedback comes from:

  • Evaluation metrics provide scalar scores that can become inputs to training. section 7.0
  • Human preference comparisons train systems without requiring people to demonstrate the correct answer from scratch, as in RLHF. section 7.0
  • Written principles become signals when an AI critic evaluates outputs against them, as in Constitutional AI. section 8.1.4
  • AI-generated labels can train a reward model in a repeated loop of candidate generation, critique, policy improvement, and auditing. section 8.2.5

Financial research assistant

To reduce invented numbers in earnings summaries, a team writes a principle forbidding facts absent from the source. An AI critic labels variants covering single-quarter questions, multi-quarter trends, and cross-company comparisons; those labels supply training feedback without collecting human rankings for every example. section 8.2.5

Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
Technique
Principled
Policy-based
Personality
RLHF
Indirect, through average labeler’s preferences
Implicit, through refusal demonstrations
Strong, through the demonstration style
RLAIF
Indirect, through the rubric the AI judge reads
Same as RLHF
Same as RLHF, plus AI judge bias
Constitutional AI
Direct, principles state values explicitly
Encodable as principles, with interpretive risk
Encodable, leaks the AI critic’s style priors

Compare how RLHF, RLAIF, and Constitutional AI turn preferences or principles into signals affecting principled behavior, policy behavior, and personality. section 8.1.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Descends from scalar-score intuition to preference, principle-based, and AI-generated feedback loops, but stops short of full reward-model implementation.
Examples
Uses RAG metrics, robot backflips, Pong, harmful-request refusals, and a financial research assistant to illustrate feedback sources.
What it assumes
Builds on scalar evaluation metrics and reinforcement-learning policy updates, while connecting them to alignment methods.
What you can do after
Apply evaluation scores, preferences, or principles as candidate training feedback while recognizing the need to test signal quality.
How it connects
Connects evaluation metrics, reinforcement learning, supervised fine-tuning, reward functions, human preferences, reward models, and Constitutional AI.

Also explained in

Compare how 4 books explain learning signal →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.