Chapter 5: Optimizing prompt examples

Introduction

Introduces DSPy prompt optimization as a data-driven search over candidate prompts and distinguishes optimizing demonstrations from optimizing instructions.

• Data-driven prompt selection: Explain how DSPy optimization uses a metric and validation dataset to choose among candidate prompts, and why it tests a manageable set rather than every possible prompt.
  Why it matters: This is the basic logic behind the optimization methods introduced in this chapter.
• Two prompt optimization targets: Distinguish optimizing the demonstrations included in a prompt from optimizing its instructions, without assuming either method has been explained here.
  Why it matters: The distinction clarifies which part of a prompt an optimization effort aims to improve.

Key terms: candidate prompts, metric function, validation dataset, data-driven, demonstrations, instructions, prompt optimization

5.1 General approaches to optimization

This section explains why labeled demonstrations can improve a classification prompt and why optimizing them means evaluating sets of examples rather than choosing appealing examples one at a time.

• Ways to improve classification prompts: Describe how clearer class names, labeled demonstrations, and more explicit instructions can help an LM distinguish intent classes with vague names such as Flight and Booking.
  Why it matters: It helps you identify what a weak prompt leaves ambiguous before trying to improve it.
• Examples versus prompt demonstrations: Distinguish the section's use of examples as dataset items from demonstrations as labeled input-output items included in a prompt.
  Why it matters: The distinction makes it clear what an optimizer selects and where the selected items go.
• Quality of the demonstration set: Explain why demonstrations should be judged as a set that teaches general patterns and covers difficult cases, rather than by whether each item seems useful on its own.
  Why it matters: A collection of individually plausible examples can still teach the wrong overall pattern.
• Practical search for demonstrations: Justify testing a manageable number of candidate demonstration sets and selecting the best-performing one instead of exhaustively testing every possible combination.
  Why it matters: It explains why prompt optimization is a measured search, not a brute-force search or a guess.
• Demonstrations as a starting point: Explain why optimizing demonstrations is often a good first move while recognizing that some subtle tasks need detailed instructions.
  Why it matters: It offers a sensible starting priority without turning it into a rule that ignores task-specific needs.

Key terms: class names, demonstrations, instructions, intent label, examples, demos, general patterns, edge cases, set as a whole, combinations

5.2 The LabeledFewShot optimizer

Introduces LabeledFewShot’s random selection of prompt examples and prepares the intent-classification program and datasets used to assess optimization.

• Random training examples in prompts: Explain that LabeledFewShot randomly selects examples from the training data and adds them to the prompt, rather than searching for the best set.
  Why it matters: This distinction sets expectations for what LabeledFewShot can improve and what it does not optimize.
• Dataset roles and scoring: Distinguish the roles of training, validation, development, and test examples, and describe how responses are scored in this setup.
  Why it matters: Keeping these roles distinct makes the reported progress and final result easier to interpret.
• Intent signature token tradeoff: Explain why the setup uses ClosedIntentSignature instead of the regular IntentSignature for efficient prompt work.
  Why it matters: Avoiding repeated labels saves input tokens and keeps generated prompts easier to inspect.
• Dataset sizes and source reversal: Describe the example counts used here and why the Hugging Face train and test labels are reversed when forming the chapter’s datasets.
  Why it matters: The source-split reversal explains how the setup obtains large development and final test sets without making the runs unnecessarily costly.

Key terms: LabeledFewShot, training data, demos, training, validation, development, test, exact matches, average score, ClosedIntentSignature

5.2.1 Executing the LabeledFewShot optimizer

This section shows how LabeledFewShot adds randomly selected training examples to an intent classifier’s prompt and how to inspect and test the resulting program.

• LabeledFewShot compilation and selection: Describe how to create a LabeledFewShot optimizer, set k, and compile the intent classifier with a training set. Explain why this optimizer does not require a validation set.
  Why it matters: This explains both what the optimizer changes and why its selected examples are not guaranteed to be the best ones.
• Demonstrations in the optimized prompt: Explain what the selected demonstrations look like in the optimized prompt and how to view them.
  Why it matters: Inspecting the prompt makes the effect of compilation visible rather than treating the optimized program as a black box.
• Testing versus evaluating compilation: Distinguish trying the compiled classifier on an utterance from evaluating whether it outperforms the original classifier.
  Why it matters: A successful sample classification does not establish that the optimized classifier performs better overall.

Key terms: LabeledFewShot, k, compile(), student, training set, validation set, demonstrations, inspect_history(), demos, User

5.2.2 Executing the LabeledFewShot optimizer repeatedly in a loop

This section shows how to search over LabeledFewShot prompts by repeating runs with different numbers of demos and using early stopping to evaluate candidates efficiently.

• Repeated random demo search: Explain why LabeledFewShot is run repeatedly, how the runs vary, and why this constitutes a random search rather than an optimizer that learns from earlier results.
  Why it matters: One run tests only one choice of demos, while repeated runs let you compare candidates and demo counts.
• Validation-based early stopping: Describe how the section screens each candidate prompt on part of the validation set before deciding whether to finish its evaluation, including the efficiency benefit and risk.
  Why it matters: Early stopping makes it practical to compare more candidate prompts without fully evaluating every weak candidate.
• More selective stopping rules: Describe the two refinements to early stopping suggested here: checking at multiple points and comparing candidates against prompts already tested.
  Why it matters: These variations can avoid full tests on more weak candidates than a single fixed-threshold check.

Key terms: LabeledFewShot, demos, k, random search, training set, early stopping, candidate prompt, validation set, fixed threshold, previously tested prompts

5.2.3 Working with ChainOfThought

This section explains what happens when a ChainOfThought program is optimized using training examples that lack reasoning fields.

• Optimizer effects on ChainOfThought: Explain that optimizing a ChainOfThought program can add demonstrations, improve instructions, or both, but does not change its module or input and output fields. Describe how its demonstrations therefore have fields for both the intent label and reasoning.
  Why it matters: It explains why optimizing this program does not remove the reasoning field from its prompt examples.
• Missing reasoning in demonstrations: Describe the options when training examples lack reasoning: add a reasoning field to each example, which may be impractical, or run the optimizer with the examples as they are. State how the resulting demonstrations mark the missing field.
  Why it matters: It lets you recognize what the optimizer puts in the prompt when your training examples have no reasoning.

Key terms: ChainOfThought, Predict, optimizer, demonstrations, intent label, reasoning, training data, reasoning field, Not supplied for this particular example

5.3 BootstrapFewShot

This section explains how BootstrapFewShot selects demonstrations for a DSPy program, how its metric governs selection, and how to recognize unsuccessful bootstrapping.

• Bootstrapping for multistep modules: Explain why bootstrapping is useful when a module makes multiple LM calls, even though its training examples provide only the module's inputs and final outputs.
  Why it matters: You can supply examples without specifying every internal step of a module.
• Labeled and bootstrapped demonstrations: Distinguish labeled demos from bootstrapped demos in the single-call airline classifier example.
  Why it matters: The two demo settings treat questionable training labels differently.
• Metric-gated demo selection: Trace how BootstrapFewShot tests candidate examples and decides which become demos in the classifier example.
  Why it matters: Bootstrapped demos depend on what the LM can predict and what the metric accepts, not just on which examples are available.
• Metrics for two purposes: Distinguish a metric's role in evaluating prompt quality from its role in deciding whether a response is good enough for a bootstrapped demo, including how a numeric metric can serve both.
  Why it matters: A metric that reports useful evaluation scores can also make the acceptance decision bootstrapping requires.
• Unsuccessful bootstrapping diagnosis: Describe what happens when no training examples pass bootstrapping and how to investigate that outcome.
  Why it matters: A compiled program may have gained no demos, so checking the optimization result matters.

Key terms: BootstrapFewShot, multistep modules, intermediate LM calls, bootstrapping, max_labeled_demos, max_bootstrapped_demos, labeled demos, bootstrapped demos, rejection sampling, metric function

5.3.1 Working with ChainOfThought

This section explains how BootstrapFewShot generates and validates ChainOfThought demonstrations, and when bootstrapping may be counterproductive.

• Bootstrapped reasoning demonstrations: Explain how BootstrapFewShot can fill a reasoning field in ChainOfThought demonstrations when max_bootstrapped_demos is greater than zero, rather than merely selecting labeled examples.
  Why it matters: It explains how a prompt can gain useful reasoning examples even when the training examples have no reasoning field.
• Metric gated demonstration content: Trace how DSPy calls the LM on training examples, uses validate_answer to accept predictions, and places the accepted prediction’s output fields into demonstrations.
  Why it matters: The metric’s decision controls which generated content enters the prompt, even when it does not inspect every field.
• When bootstrapping helps or hurts: Justify when to enable bootstrapped demonstrations and when to set max_bootstrapped_demos to zero for the airline-classifier situation.
  Why it matters: More filtering is not automatically better if it removes the hard examples the prompt needs.

Key terms: BootstrapFewShot, ChainOfThought, max_labeled_demos, max_bootstrapped_demos, reasoning field, bootstrapping, validate_answer, intent_label, metric function, prediction

5.3.2 Specifying a teacher

This subsection explains why a stronger teacher LM can be used during bootstrapping and illustrates it with GPT-4 generating ChainOfThought reasoning fields.

• Teacher LM in bootstrapping: Explain that a teacher LM lets a stronger LM generate content during optimization, which can help with bootstrapping but is not always beneficial. Describe the example of GPT-4 generating ChainOfThought reasoning fields.
  Why it matters: It clarifies what the teacher contributes to bootstrapping without assuming that a stronger LM always improves the result.

Key terms: teacher LM, bootstrapping, ChainOfThought, reasoning fields

5.4 BootstrapFewShotWithRandomSearch

The section explains how BootstrapFewShotWithRandomSearch compares demonstration sets on validation data, how to configure and interpret the search, and why the number of demos is worth testing.

• Validation-Based Demo Set Selection: Explain how BootstrapFewShotWithRandomSearch improves on selecting examples at random once: it evaluates multiple candidate sets of demonstrations and selects the one that performs best on a validation set.
  Why it matters: Trying and scoring several demo sets gives you a way to improve on a single fast, random choice.
• Search Configuration and Validation: Describe what the candidate-count, validation-set, and thread settings control, including what happens when no validation set is supplied and the tradeoff in using a larger one.
  Why it matters: These settings determine how broadly candidates are tested and how much time and cost the comparison takes.
• Candidate Scores and Baseline: Explain why the example reports more scores than the requested number of random candidates, and how the no-demo baseline helps interpret those scores.
  Why it matters: The baseline and extra cases prevent you from mistaking every tested demo set for an improvement.
• Choosing the Demo Count: Justify testing different numbers of demonstrations rather than assuming that more demos always improve accuracy.
  Why it matters: Searching for good examples cannot by itself tell you whether you chose a good number of examples.

Key terms: BootstrapFewShotWithRandomSearch, demos, validation set, num_candidate_programs, num_threads, compile(), baseline, LabeledFewShot, BootstrapFewShot, max_bootstrapped_demos

5.5 KNN: Finding examples dynamically

This section explains how DSPy’s KNN optimizer selects similar training examples for each LM call and what affects that approach’s usefulness and cost.

• Per-call demonstration selection: Explain how KNN optimization differs from choosing one fixed set of demonstrations: it prepares training examples at compile time but selects the k most similar ones when each LM call is made.
  Why it matters: You can explain why the prompt examples change with the request instead of having to cover every possible request at once.
• Embedding-based neighbor matching: Trace how the program represents training examples and a new input as embeddings, compares them for similarity, and puts the nearest examples into the prompt.
  Why it matters: This connects the idea of a relevant example to the actual matching step the program performs.
• Neighbor selection tradeoffs: Describe how the choice of k, the training set, and the embedder affect the quality and cost of KNN-selected demonstrations.
  Why it matters: These tradeoffs explain why choosing more examples or a larger embedder is not automatically better.
• Neighbor voting versus LM classification: Distinguish using the neighbors’ labels directly to classify an utterance from using those neighbors as demonstrations for an LM, and explain when the LM may help.
  Why it matters: Finding close examples does not mean their labels must make the final decision.

Key terms: KNN optimizer, demonstrations, k-nearest neighbors, compile(), embeddings, vector similarity, embedder, k, training set, dimensions

5.6 Evaluation results

This section explains how to compare optimized prompts, why optimizer choice depends on the task, and why small gains on an already strong classifier need careful evaluation.

• Best prompt over best optimizer: Explain why there is no universally best optimizer and why the practical choice is the best-performing prompt, regardless of which optimizer produced it.
  Why it matters: It keeps the decision about what to use in production tied to observed prompt performance.
• Random and per-prompt demonstrations: Distinguish the first three optimizers’ random selection of demonstration sets from KNN’s nonrandom selection of different demonstrations for each prompt.
  Why it matters: The optimizers being compared do not all choose examples in the same way.
• Interpreting small classification gains: Explain why gains over this classifier’s roughly 91% unoptimized score may be small and why apparent differences between prompts need careful measurement.
  Why it matters: A small measured improvement matters only if the comparison is reliable enough to support it.

Key terms: optimizer, prompt, demonstrations, bootstrapping, BootstrapFewShotWithRandomSearch, KNN, classification, unoptimized module, random variation, evaluation framework

5.6.1 gpt-4o-mini

This section interprets gpt-4o-mini optimization results and draws practical lessons about demonstration selection, hyperparameter testing, and inference tradeoffs.

• Few-shot optimization results: Explain how the tested optimizers performed against the unoptimized baselines and why the results support trying many demonstration sets.
  Why it matters: A single randomly selected set of examples may not reveal how well optimization can work.
• Hyperparameter search strategy: Describe how to choose initial hyperparameter values and narrow a follow-up search using the LabeledFewShot k example.
  Why it matters: A broad first pass gives a sensible starting point for more focused experiments.
• Optimized module comparison: State how ChainOfThought compared with Predict after optimization and how that affected the experiments reported.
  Why it matters: This explains why the reported experiments emphasize ChainOfThought.
• KNN accuracy tradeoff: Explain the accuracy and resource tradeoff reported for KNN using fewer examples.
  Why it matters: The highest accuracy is not the only consideration when choosing an optimization approach.

Key terms: gpt-4o-mini, LabeledFewShot, baseline, demos, demonstrations, hyperparameters, k value, ChainOfThought, Predict, KNN

5.6.2 gpt-4.1-nano

This subsection shows that optimizing prompts can make the cheaper, initially weaker gpt-4.1-nano competitive with gpt-4o-mini.

• Optimized nano cost tradeoff: Explain why gpt-4.1-nano becomes a reasonable lower-cost choice after optimization, despite its weaker unoptimized performance.
  Why it matters: Optimization can change which model offers the best balance of performance and cost.

Key terms: gpt-4.1-nano, gpt-4o-mini, unoptimized DSPy programs, optimized programs

5.7 Summary