liveBook · Manning

Ground-truth signal

As explained in Self-Improving Agents

Feedback that evaluates an output by comparing it with a known correct answer.

A ground-truth signal scores an agent’s output against a known correct answer. If an agent must produce a revenue leaderboard and the correct ranking is already known, the answer can be checked rather than sent to a model judge for an opinion. The same answer then receives the same score each time. section 1.4.1 section 3.2.2

Reach for ground truth when a reliable answer key exists. It is grounded outside the model’s opinion, but labelled answers can be expensive, become stale, or simply not exist. An environment signal offers another external check, such as a unit test result; a judge-based signal is easier to obtain for open-ended work but is less precise. section 1.4.1

Check the checker before searching

Run two prompts through the same revenue agent and scorer. A first, sparse prompt establishes a baseline. A known-good reference prompt spells out the rules used to derive the correct leaderboard and should score perfectly. If it does not, investigate the task or checker before using its scores to choose improvements. Even a deterministic checker is code that can be wrong; its verdict is a measurement, not a proof. section 3.2.2 section 3.6

Listing 3.3 Check the checker (`02_revenue_check.py`)
from agents.revenue import (
    POLICY_GENESIS, POLICY_ORACLE, TASK_REQUEST, build_revenue_agent,
    reconstruct_answer, score_revenue,
)
 
async def run(policy):
    agent = build_revenue_agent(policy, temperature=0.0)    #A
    _, traj = await agent.run(TASK_REQUEST)
    return score_revenue(reconstruct_answer(traj))    #B
 
async def main():
    for label, policy in [("GENESIS", POLICY_GENESIS),
                          ("ORACLE", POLICY_ORACLE)]:    #C
        for i in range(2):
            score = await run(policy)
            print(f"{label} run{i + 1}: score={score:.2f}")

The Python listing runs both prompts through the agent, reconstructs each answer, and applies the same revenue scorer twice to each. section 3.2.2

Use a strict exact-match check to validate the answer key and checker. For a search across candidate prompts, a smoother score can show partial progress where exact match gives only a verdict. Neither helps if the checker rewards the wrong answer. section 3.6 section 3.2.2

How Self-Improving Agents covers it

How deep it goes
Defines the comparison and works through checker calibration, while stopping short of treating a deterministic score as proof.
Examples
Uses a revenue leaderboard, a sparse baseline prompt, a known-good reference prompt, and Python runs through a shared scorer.
What you can do after
Diagnose/debug with it by checking a scorer against baseline and known-good outputs before trusting it to rank candidates.
How it connects
Connects ground truth to environment signals, model judges, exact-match checks, smooth scoring functions, and improvement loops.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.