Chapter 2: Getting started improving agents
The chapter introduces an offline improvement loop for a RAG agent’s prompt instruction and previews the tools used to build it.
After this chapter you will be able to
- Prompt instruction as improvement artifact. Explain why this example focuses on improving the RAG agent's prompt instruction rather than the whole platform.
- Practical LLM model choice. Justify choosing the cheapest, fastest model that can do the job for most agent applications, while recognizing when advanced reasoning may warrant a cutting-edge model.
- Purpose of the Helix framework. Explain why the book uses Helix for self-improving agents instead of relying on a typical agent framework.
- Basic Helix RAG agent. Describe what is configured in the basic Helix RAG agent and what Helix supplies when it runs.
- Artifact selection and promotion. Explain how the archive retains prompt variants and their measured signals, how best() identifies the top-scoring artifact, and why applying it requires an explicit promote() call.
- Initial archived prompt setup. Describe how this version of the RAG agent attaches its system prompt to a SQLite-backed archive, marks the initial prompt as the genesis artifact, and obtains its prompt from the archive rather than a direct text string.
- Standard behind the signal. Explain why the improvement loop needs a clear standard for a good response before it can produce a meaningful evaluation.
- Precision and availability tradeoff. Explain precision and availability as the two axes for choosing a signal, and justify choosing the most precise signal that is practical for the task rather than an absolute best signal.
- Signal inputs and measurement. Explain that a signal takes the agent’s output and a standard for what good looks like, then measures the gap between them.
- Fixed GapMeasurement output. Describe the common GapMeasurement returned by signal methods and explain why its fixed shape keeps Search and the archive independent of the measurement method.
- Swap-and-agree position bias check. Explain how comparing the same two answers in both orders detects position bias, and describe how agreement or disagreement determines the verdict.
- Ordered judge rubric. Describe what the rubric tells the judge, how this section prioritizes its criteria, and why its placement and ordering matter.
- Search for better candidates. Explain that search looks for a better path by selecting a candidate from an archive or creating a new one, rather than looking up an existing answer.
- Search in the improvement loop. Explain why a signal alone cannot improve an answer and how search uses it to find a better system prompt. Describe the LLM's role in generating candidates and the judge's role in testing them.
- Shared search method interface. Describe the inputs, two moves, and output shared by search methods in this section. Explain why the selected prompt is a new artifact version with a link to its parent.
- Search cost and reach. Explain what cost and reach mean on the search-method map and describe the general trade-off illustrated by SPO, GEPA, and DGM.
- Cheapest capable search rule. Justify choosing the cheapest search method that can reach the solution, rather than automatically choosing the method with the greatest reach.
- Repeated search cycle. Trace how search repeats the self-improvement cycle: propose a variant of the current best artifact, measure it with a signal, and keep it only if it scores better. Explain that each round starts from the best candidate so far and the survivor becomes the new artifact.
- Pluggable proposal step. Distinguish the part of search that changes between methods from the parts that stay the same: methods differ in how Propose makes variants, while Measure and Select still score candidates and keep the winner.
- Comparison Based Self Supervision. Explain how SPO obtains a signal without a labeled answer key: the current prompt and a challenger answer the same questions, and a pairwise judge prefers one set of outputs.
- Mutate Score Accept Loop. Trace one SPO round: mutate the current best prompt using feedback, score it against a challenger on sample questions with a pairwise judge, and accept the candidate only if it wins more questions than it loses.
- SPO prompt improvement loop. Trace how the three functions implement SPO: propose_mutation rewrites a prompt using feedback, score compares the candidate with a reference and counts wins, and SPO accepts a candidate only when it wins.
- Self-supervised improvement signal. Explain how SPO search and a pairwise judging signal improved the example prompt without human editing or a correct answer supplied to the judge.
- Helix loop composition. Explain why assembling the self-improvement loop with Helix does not require rebuilding its plumbing: Helix provides the previously hand-built pieces as objects to compose.
- Offline improvement components. Describe how the Signal and Search compose into an OfflineImprover, and where it runs and stores its results.
- Live deployment boundary. Explain why offline candidates do not automatically change the live agent and how a winner reaches production.
- Offline improvement round. Trace how OfflineImprover proposes a prompt candidate, measures it against the current best on eval questions, and records the result, including what Policy governs and why the round is offline.
- Candidate testing isolation. Explain how OfflineImprover tests a candidate prompt without mutating the agent definition passed to it.
- Improvement round scheduling. Explain that the ImproverPolicy is passed to the OfflineImprover once and consulted each round to control scheduling. Distinguish MANUAL from INTERVAL and CONTINUOUS scheduling.
- Candidate promotion gate. Explain what the candidate win-rate threshold determines and why clearing it does not automatically promote a candidate in OFFLINE mode.
- Helix script composition. Trace the main steps of main_async and explain how the framework replaces the hand-written improvement loop with a small amount of setup code.
- Offline promotion gate. Explain what the three rounds change, what the agent continues to serve, and how a candidate becomes the served prompt.
What it builds on
- What an artifact is (chapter 1) · Introduces what an improvement artifact is.
- The loop, walked through one task (chapter 1) · The idea of an improvement loop.
- The signal family (chapter 1) · The earlier introduction to signals in the improvement loop.
Section by section
Introduction
The chapter introduces an offline improvement loop for a RAG agent’s prompt instruction and previews the tools used to build it.
2.1 Building the agent and text prompt artifact
This introduction previews the agent setup, model choices, and archive needed before improvement can begin.
2.1.1 Introducing the RAG agent
Introduces a RAG agent as the example for improving a prompt instruction, and explains why the example can work with different document collections.
- RAG knowledge augmentation: Explain why teams commonly use a RAG agent: it lets them augment an agent with their own knowledge without retraining it.
Why it matters: This explains why a RAG agent is a useful starting example. - Prompt instruction as improvement artifact: Explain why this example focuses on improving the RAG agent's prompt instruction rather than the whole platform.
Why it matters: It keeps the improvement target distinct from the many other parts of a RAG platform. - Reusable document corpus: Describe how the example's document collection relates to the Helix agent platform.
Why it matters: The supplied papers are a starting collection, not a limit on where the agent can be used.
Key terms: RAG (Retrieval Augmented Generation) agent, prompt instruction, prompt artifact, self-improving agent, Helix, document corpus
2.1.2 Running the code and picking LLMs
This section explains the LLM access needed to run the examples and why the book usually favors inexpensive, available models over cutting-edge ones.
- Practical LLM model choice: Justify choosing the cheapest, fastest model that can do the job for most agent applications, while recognizing when advanced reasoning may warrant a cutting-edge model.
Why it matters: Model choice affects whether repeated improvement runs are affordable and practical. - LLM access and deployment options: State what access is needed to run the source code and describe the commercial and open-weight options mentioned, including why local deployment can be useful.
Why it matters: You can choose an accessible setup that suits repeated experiments and your data constraints.
Key terms: improvement loop, signal and search methods, rate limits, cutting-edge model, LLM API, commercial service, open-weight models, local/cloud options
2.1.3 Discovering layer 0 – the helix framework
This section explains why Helix is used and how its basic RAG agent is assembled and run.
- Purpose of the Helix framework: Explain why the book uses Helix for self-improving agents instead of relying on a typical agent framework.
Why it matters: It explains why the examples use Helix rather than retrofit another agent framework. - Basic Helix RAG agent: Describe what is configured in the basic Helix RAG agent and what Helix supplies when it runs.
Why it matters: It separates what the example configures from the retrieval setup and execution loop Helix handles. - Example corpus and replacement: State what corpus the base RAG agent uses and what changing it entails.
Why it matters: It identifies the documents behind the book's RAG examples and the stated way to replace them.
Key terms: Helix, hooks, tracking, improvement patterns, RAG agent, build_retrieval_tool(), harness, model, prompt, tools
2.1.4 Connecting the prompt artifact archive
The section connects a RAG agent’s system prompt to an artifact archive and explains how the archive records, selects, and promotes prompt versions.
- Artifact selection and promotion: Explain how the archive retains prompt variants and their measured signals, how best() identifies the top-scoring artifact, and why applying it requires an explicit promote() call.
Why it matters: It separates finding a promising prompt from deciding to use it. - Initial archived prompt setup: Describe how this version of the RAG agent attaches its system prompt to a SQLite-backed archive, marks the initial prompt as the genesis artifact, and obtains its prompt from the archive rather than a direct text string.
Why it matters: The agent can now obtain its prompt through the archive even before any improvement has been made.
Key terms: artifact archive, measured signals, best(), promote(), SQLite archive, system prompt, genesis, prompt Artifact
2.2 Measuring the signal
The section explains why measuring an agent’s performance starts with defining what a good response contains.
- Standard behind the signal: Explain why the improvement loop needs a clear standard for a good response before it can produce a meaningful evaluation.
Why it matters: A clear standard lets the loop judge whether a response is good rather than produce an unhelpfully vague verdict.
Key terms: signal, standard, evaluation
2.2.1 Establishing what good is
This section explains ways to judge an agent’s output and how to choose an evaluation signal by balancing precision against availability.
- Forms of evaluation signal: Describe how a gold set, an LLM-as-Judge rubric, a programmatic check, or a human evaluator can establish what good output looks like and measure the gap between that standard and the agent’s output.
Why it matters: You need a standard for good output before you can tell whether an agent should improve. - Precision and availability tradeoff: Explain precision and availability as the two axes for choosing a signal, and justify choosing the most precise signal that is practical for the task rather than an absolute best signal.
Why it matters: A useful evaluation signal must both reflect quality and be feasible to obtain.
Key terms: gold set, rubric, LLM-as-Judge, programmatic, human evaluator, signal, precision, availability
2.2.2 Producing the signal gap measurement
This section defines the signal as an interchangeable way to measure the gap between an agent’s output and a standard of good, while keeping its inputs and output format fixed.
- Signal inputs and measurement: Explain that a signal takes the agent’s output and a standard for what good looks like, then measures the gap between them.
Why it matters: This lets you change how you evaluate an agent without rebuilding the improvement loop. - Fixed GapMeasurement output: Describe the common GapMeasurement returned by signal methods and explain why its fixed shape keeps Search and the archive independent of the measurement method.
Why it matters: A consistent result lets later parts of the loop use the measurement without depending on how it was made.
Key terms: signal, agent's output, gap, LLM-as-Judge, GapMeasurement, score, preference, feedback, confidence, Search
2.2.3 Implementing LLM-as-Judge for signal measurement
This section shows how a pairwise LLM judge uses an ordered rubric and a swap-and-agree check to produce a more reliable comparison signal.
- Swap-and-agree position bias check: Explain how comparing the same two answers in both orders detects position bias, and describe how agreement or disagreement determines the verdict.
Why it matters: The swap check prevents a preference for the first answer from masquerading as a quality judgment. - Ordered judge rubric: Describe what the rubric tells the judge, how this section prioritizes its criteria, and why its placement and ordering matter.
Why it matters: A fixed, prioritized rubric makes comparisons reflect the task’s standards rather than the judge’s vague preferences. - Structured comparison signal: State what inputs the pairwise judge receives and what structured signal its comparison produces.
Why it matters: A structured verdict lets code store and use the comparison instead of parsing an open-ended critique.
Key terms: LLM-as-Judge, pairwise evaluation, position bias, swap-and-agree, TIE, rubric, system prompt, reference answer, factual accuracy, grounded in the sources
2.3 Searching for artifact variants
Search seeks better artifact candidates for self-improvement rather than retrieving an answer the agent already has.
- Search for better candidates: Explain that search looks for a better path by selecting a candidate from an archive or creating a new one, rather than looking up an existing answer.
Why it matters: This distinction clarifies what search contributes to improving an agent.
Key terms: Search, candidate, archive
2.3.1 Selecting the search method
This section explains how search uses a measured signal to propose and select improved prompt artifacts, and why the chapter starts with a simple search method.
- Search in the improvement loop: Explain why a signal alone cannot improve an answer and how search uses it to find a better system prompt. Describe the LLM's role in generating candidates and the judge's role in testing them.
Why it matters: It separates measuring performance from making a change that might improve it. - Shared search method interface: Describe the inputs, two moves, and output shared by search methods in this section. Explain why the selected prompt is a new artifact version with a link to its parent.
Why it matters: The common interface allows different search methods to produce artifacts that the rest of the improvement loop can handle the same way. - Starting search method choice: Justify starting with the cheapest search method that improves the measurement and switching to a more elaborate method only when that approach stalls.
Why it matters: A more elaborate search is not the default when a simpler one already improves the measured result.
Key terms: signal, search, system prompt, artifact, judge, current best artifact, live-champion prompt, variants, parent pointer, lineage
2.3.2 Search cost versus reach
This section maps search methods by cost and reach, then gives a rule for choosing a method without paying for unnecessary power.
- Search cost and reach: Explain what cost and reach mean on the search-method map and describe the general trade-off illustrated by SPO, GEPA, and DGM.
Why it matters: The map makes the trade-off between search expense and capability explicit. - Cheapest capable search rule: Justify choosing the cheapest search method that can reach the solution, rather than automatically choosing the method with the greatest reach.
Why it matters: It prevents spending extra compute on search power that the task does not require.
Key terms: cost, reach, rising frontier, SPO, GEPA, DGM, search, landscape, cheapest search
2.3.3 Inside the search
Search repeatedly proposes, measures, and selects artifact variants, with search methods differing in how they propose variants.
- Repeated search cycle: Trace how search repeats the self-improvement cycle: propose a variant of the current best artifact, measure it with a signal, and keep it only if it scores better. Explain that each round starts from the best candidate so far and the survivor becomes the new artifact.
Why it matters: This lets you follow what a search run does to an artifact from one round to the next. - Pluggable proposal step: Distinguish the part of search that changes between methods from the parts that stay the same: methods differ in how Propose makes variants, while Measure and Select still score candidates and keep the winner.
Why it matters: It gives you a way to compare search methods without confusing their proposal strategies with the rest of the loop.
Key terms: Propose, Measure, Select, signal, artifact, pluggable slot, SPO, GEPA
2.3.4 Understanding SPO (Self-Supervised Prompt Optimization)
Explains how SPO improves a prompt by comparing outputs rather than grading them against labeled answers, and why that makes the search cheap but limited.
- Comparison Based Self Supervision: Explain how SPO obtains a signal without a labeled answer key: the current prompt and a challenger answer the same questions, and a pairwise judge prefers one set of outputs.
Why it matters: This is why SPO can optimize prompts without the work of preparing an answer key. - Mutate Score Accept Loop: Trace one SPO round: mutate the current best prompt using feedback, score it against a challenger on sample questions with a pairwise judge, and accept the candidate only if it wins more questions than it loses.
Why it matters: The acceptance rule explains how judged preferences steer each successive prompt revision. - Low Cost And Plateau: Explain why SPO can be cheap and why its hill climb can stop: comparisons can use as few as three sample questions per round, but progress plateaus when no candidate beats the current best.
Why it matters: SPO’s low cost comes with a clear limit on how far this search can climb.
Key terms: Self-Supervised Prompt Optimization, self-supervision, pairwise judge, labeled answer key, Mutate, Score, Accept, hill climb, current best, sample questions
2.3.5 Implementing SPO search
This section shows how a small SPO implementation proposes, scores, and accepts prompt improvements using a pairwise judge without correct answers.
- SPO prompt improvement loop: Trace how the three functions implement SPO: propose_mutation rewrites a prompt using feedback, score compares the candidate with a reference and counts wins, and SPO accepts a candidate only when it wins.
Why it matters: These three steps explain how the demo improves a prompt without a person editing each variant. - Proposer and judge rubrics: Explain why the proposer generally receives the judge's rubric but may need some rules omitted, using grounding as the example.
Why it matters: Separating grading criteria from writing instructions helps make the proposer's guidance usable. - Self-supervised improvement signal: Explain how SPO search and a pairwise judging signal improved the example prompt without human editing or a correct answer supplied to the judge.
Why it matters: This shows how a prompt can improve even when the judge has no reference answer.
Key terms: propose_mutation, score, SPO, mutate, score, accept, swap-and-agree judge, proposer, rubric, judge, grounding, pairwise judge
2.4 Putting it together with Helix
Helix packages the self-improvement loop’s previously hand-built pieces as objects that can be composed instead of rebuilt.
- Helix loop composition: Explain why assembling the self-improvement loop with Helix does not require rebuilding its plumbing: Helix provides the previously hand-built pieces as objects to compose.
Why it matters: This connects the loop you built by hand to the Helix components you will use next.
Key terms: Helix, artifact in an archive, pairwise judge, SPO search, primitives
2.4.1 Understanding the helix components
This section shows how Helix combines offline evaluation and search while keeping changes to the live agent behind a deliberate promotion gate.
- Offline improvement components: Describe how the Signal and Search compose into an OfflineImprover, and where it runs and stores its results.
Why it matters: This explains where candidate improvements come from without changing the serving agent. - Live deployment boundary: Explain why offline candidates do not automatically change the live agent and how a winner reaches production.
Why it matters: The promotion gate lets improvement run without automatically deploying unapproved candidates.
Key terms: Signal, Search, SPO, OfflineImprover, SQLiteArchive, eval set, Live, archive, deploy gate, promote()
2.4.2 The OfflineImprover
This section shows how OfflineImprover runs one offline improvement round and tests a candidate prompt without changing the original agent.
- OfflineImprover component inputs: Describe the six ingredients given to OfflineImprover: the Agent, the ID of the artifact to improve, and the Search, Signal, Archive, and Policy primitives.
Why it matters: These inputs connect the agent and its chosen artifact to the improvement loop. - Offline improvement round: Trace how OfflineImprover proposes a prompt candidate, measures it against the current best on eval questions, and records the result, including what Policy governs and why the round is offline.
Why it matters: The round separates trying and scoring a prompt from promoting it or using it on live traffic. - Candidate testing isolation: Explain how OfflineImprover tests a candidate prompt without mutating the agent definition passed to it.
Why it matters: Testing a proposed prompt in a copy preserves the original agent definition.
Key terms: OfflineImprover, Agent, artifact ID, Search, Signal, Archive, Propose, Measure, Record, agent.with_artifacts(candidate)
2.4.3 The ImproverPolicy
The ImproverPolicy configures when improvement rounds run, how much they can cost, and when a candidate qualifies as a winner.
- Improvement round scheduling: Explain that the ImproverPolicy is passed to the OfflineImprover once and consulted each round to control scheduling. Distinguish MANUAL from INTERVAL and CONTINUOUS scheduling.
Why it matters: The schedule determines whether you start improvement rounds yourself or let the improver run them. - Per-round cost controls: Describe how the ImproverPolicy limits the size and cost of each improvement round.
Why it matters: These settings keep both a whole round and an individual evaluation question within limits. - Candidate promotion gate: Explain what the candidate win-rate threshold determines and why clearing it does not automatically promote a candidate in OFFLINE mode.
Why it matters: A qualifying score and an actual promotion are separate decisions in OFFLINE mode.
Key terms: ImproverPolicy, OfflineImprover, MANUAL, INTERVAL, CONTINUOUS, trigger_round(), eval questions, budget, per-question caps, promote_threshold_win_rate
2.4.4 The code
This section shows how a short Helix script runs offline prompt improvement while keeping measured candidates separate from the prompt the agent serves.
- Helix script composition: Trace the main steps of main_async and explain how the framework replaces the hand-written improvement loop with a small amount of setup code.
Why it matters: The small script shows where the improvement parts are assembled and what Helix handles for you. - Offline promotion gate: Explain what the three rounds change, what the agent continues to serve, and how a candidate becomes the served prompt.
Why it matters: The promotion gate lets you inspect improvement results before they affect the agent's behavior.
Key terms: main_async, archive, seed, four primitives, improver, rounds, offline improvement, measured candidates, genesis prompt, deploy gate
2.5 Exercises
Five exercises have the reader establish a RAG-agent baseline, verify its prompt archive, test judge biases, and promote an improved prompt.
2.6 Summary
This summary brings together how an offline loop improves a RAG agent’s prompt by measuring outputs, searching prompt variants, and deliberately promoting a winner.
Where this is used later
- (chapter intro) (chapter 3) · Earlier work on measuring an agent’s signal.
- Measuring the gap (chapter 3) · Introduces establishing what good looks like.
- Consuming signals with search (chapter 3) · Measured signals used to compare candidates.
- (chapter intro) (chapter 4) · Introduces search cost as a consideration.
- A deceptive landscape: greedy sticks, an archive crosses (chapter 4) · A deceptive landscape provides a case where greater search reach may matter.
- Introducing Genetic-Pareto evolutionary search (chapter 4) · Introduces GEPA as another search approach to compare with the generic cycle.
- SPO: climbing without a ground truth (chapter 3) · The discussion of SPO revisits climbing without ground truth.
Concepts explained in this chapter
- LLM-as-a-judgeA pattern in which a large language model evaluates outputs against a rubric, potentially comparing candidates to select the best.
- swap-and-agreeA pairwise judging technique that evaluates answers in both possible orders and accepts a verdict only when the judgments agree.
- GapMeasurementA structured evaluation result that records an assessment of the gap between a candidate output and a desired standard, along with supporting feedback.
- position biasA systematic influence of an answer's position in a comparison on a judge's evaluation, independent of answer quality.
- SPOSPO is a pairwise search method that generates and compares candidate variants.
- searchSearch is an optimization process that generates candidate artifact variants and selects among them according to a signal.
- rubricA set of criteria for evaluating an output.
- promptText supplied to a language model as instructions or input for an interaction.
- offline loopAn agent-improvement cycle that evaluates proposed changes and applies them after a deployment review.
- availabilityThe ease and speed with which a signal can be obtained for a task, considering cost and required resources.
- precisionThe accuracy with which a signal reports the size of the quality gap between an output and the desired result.
- archiveAn archive is a stored collection of past candidates retained during a search process.
- measure-change-measure cycleA recurring process of measuring an agent's output, finding an improved version, and submitting it.
- self-improving agentAn agent whose capabilities improve through changes to the shared instructions, tools, or other infrastructure around its model rather than changes to the model itself.
- hill climbingAn iterative search method that makes local changes to a candidate and retains changes that improve its score.
- evolutionary computationA search method that improves candidate solutions through variation and selection across iterations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
