Chapter 8: Summarization and more effective metric functions

Introduction

This introduction explains why evaluating open-ended LM outputs, especially summaries, requires more sophisticated metric functions than evaluating simple classification-style answers.

• Simple versus open-ended evaluation: Distinguish tasks whose outputs can be checked with a simple correctness metric from tasks, such as summarization, that require a more complex metric.
  Why it matters: The kind of output determines whether a simple correctness check is enough.
• Multifaceted output quality metrics: Explain what a metric for an open-ended LM response must capture and who is responsible for defining it in a DSPy application.
  Why it matters: Reliable evaluation and optimization depend on a score that reflects what makes an output useful for the application.

Key terms: metric functions, text summarization, entailment, open-ended question answering, accuracy, relevance, completeness, wording, numeric score

8.1 Summarization

This section explains why judging a document summary requires balancing several qualities, while affirming that meaningful evaluation is possible.

• Summary quality considerations: Explain why judging a document summary is difficult: it may omit important content, contain incorrect statements, include unnecessary details, or have problems with phrasing or level of detail. State that these considerations can be hard to balance but summaries can still be evaluated meaningfully.
  Why it matters: These competing qualities explain why evaluating summaries takes more than a quick judgment of whether they sound good.

Key terms: document summary, important content, level of detail

8.1.1 Types of summarization

This section distinguishes extractive from abstractive summarization and explains their different readability and accuracy challenges.

• Two summarization approaches: Distinguish extractive summaries, which assemble passages from the source, from abstractive summaries, whose wording is determined by the LM and need not match the source.
  Why it matters: The distinction explains why a good summary does not always reuse the source’s words.
• Extractive summary challenges: Explain why selecting source passages for an extractive summary requires attention to relevance, brevity, coherence, and readability.
  Why it matters: Taking accurate snippets from a source does not automatically make a readable summary.
• Abstractive readability and accuracy: Describe the readability advantage and accuracy risk of abstractive summaries, and state why they have become more common.
  Why it matters: A fluent summary still needs to be checked for accuracy.

Key terms: extractive summarization, abstractive summarization, relevant passages, coherent, readable, accuracy, large LMs

8.1.2 Evaluating summarization

This section explains how to choose and assess summary-evaluation methods, including when examples have no ground truth summary.

• Ground Truth Availability: Distinguish evaluation with ground truth summaries from evaluation without them. Explain that DSPy can use examples containing only the original text, but without a reference summary, evaluation must compare the generated summary with that text using LLM-as-a-judge methods.
  Why it matters: You can evaluate a summarizer even when creating reference summaries is impractical.
• Task Appropriate Metric: Explain why a summarization metric must reflect the project's criteria for a good summary rather than rely on exact matches or a universal standard.
  Why it matters: A metric can reward the wrong output if it measures something other than the qualities your project needs.
• Lexical Comparison Tradeoffs: Describe what lexical comparison measures and explain both its practical appeal and its limits for summary evaluation.
  Why it matters: Word overlap is easy to use, but it can misgrade summaries that paraphrase or alter a statement's meaning.
• Semantic Summary Comparison: Describe how embedding comparison evaluates reference and generated summaries, and distinguish semantic comparison from matching their wording.
  Why it matters: Semantic comparison can recognize a sound paraphrase that word-overlap scoring might miss.
• LLM Judge Capabilities: Explain what an LLM judge can be instructed to assess, why using one LM to judge another is not necessarily circular, and where this approach can be less suitable.
  Why it matters: A judge lets you state the qualities that matter, while its limitations help you decide when to use it.
• Multiple Plausible References: Justify including several good reference summaries for one original text and describe how they are used in evaluation.
  Why it matters: Several valid references reduce the chance of penalizing a summary merely for choosing different reasonable points.

Key terms: ground truth summaries, unlabeled data, LLM-as-a-judge, metric function, lexical evaluations, embeddings, lexical comparison, extractive summaries, abstractive summaries, text embeddings

8.2 The DialogSum dataset

The section introduces DialogSum’s human-written conversation summaries and prepares separate datasets for optimizing and evaluating a summarizer.

• DialogSum summary guidelines: Describe the guidelines used to write DialogSum summaries: retain the most salient information and important names while being brief, formal, and written from an observer’s perspective.
  Why it matters: These guidelines clarify what the dataset treats as a useful conversation summary.
• Abstractive single reference summaries: Explain that DialogSum supplies one human-written summary per dialogue and that its summaries can use different words from the dialogue while retaining its main ideas. Describe why another, somewhat different summary could also be good.
  Why it matters: A generated summary can express the right ideas without copying the dialogue or matching the reference word for word.
• Summarizer dataset split purposes: Distinguish the train, validation, and test sets created for the summarizer, including their sizes and their roles in optimization versus final evaluation.
  Why it matters: Keeping a separate test set preserves a distinct final evaluation after optimization.

Key terms: DialogSum, salient information, named entities, observer perspective, abstractive, dialogue, summary, train set, validation set, test set

8.3 Baseline summarizer

The section builds a minimal DSPy dialog summarizer, considers adding annotation instructions, and checks its output against one ground-truth summary.

• Minimal baseline summarizer: Describe how the baseline summarizer is assembled and tried on a data row: it uses gpt-4.1-nano, a minimal one-input/one-output signature, and the Predict module.
  Why it matters: This gives you a simple starting point to compare with later changes.
• Optional annotation instructions: Explain why the annotators’ directions could be added to the signature and why the initial baseline omits them.
  Why it matters: It clarifies that the bare-bones signature is a starting choice, not the only way to guide summarization.
• Initial output inspection: Describe what the example comparison and repeated executions show about judging the baseline by eye.
  Why it matters: An appealing example is useful, but it can hide inconsistent output length.

Key terms: baseline model, signature, Predict, gpt-4.1-nano, annotators, ground truth summaries, docstring, ground truth summary, LM’s summary

8.4 Evaluation by lexical similarity

The section explains how to evaluate generated summaries by comparing their wording with provided summaries, using Jaccard similarity and n-grams.

• Jaccard similarity for summaries: Explain how Jaccard similarity measures word overlap between a generated summary and a provided summary, and interpret a low score.
  Why it matters: It gives you a concrete way to score how closely a generated summary matches the wording of a reference.
• N-gram overlap tradeoff: Describe how comparing n-grams differs from comparing individual words and why longer matching sequences can be more meaningful.
  Why it matters: It helps you choose whether a lexical comparison should reward isolated shared words or shared phrases.

Key terms: lexical metric function, Jaccard similarity score, provided summaries, n-grams, 2-grams, word level

8.4.1 Precision and recall

This section defines word-overlap precision and recall for summaries, explains one way to combine them, and identifies limits of unique-word comparisons.

• Summary precision and recall: Distinguish precision from recall when comparing an LM-generated summary with a ground truth summary: precision looks at the generated summary’s words, while recall looks at the ground truth summary’s words.
  Why it matters: The two measures show different sides of word overlap between a generated summary and its reference.
• Balancing the two measures: Explain why the example combines precision and recall by taking their minimum rather than their average.
  Why it matters: The choice of combination determines whether a weak measure can be offset by a strong one.
• Unique-word comparison limits: Describe what the discussed Jaccard similarity, precision, and recall tests leave out, and how checking n-grams can partly address those omissions.
  Why it matters: Word-overlap scores do not capture every difference between summaries.

Key terms: precision, recall, LM’s summary, ground truth summary, average, minimum, Jaccard similarity, unique words, word order, n-grams

8.4.2 The F1 score

This section explains how F1 combines precision and recall, how DSPy normalizes text before calculating it, and how to evaluate a summarizer across a test set.

• F1 score and balance: Explain that F1 is the harmonic mean of two metrics, usually precision and recall, and why it reflects a low value more strongly than an arithmetic mean does.
  Why it matters: A single F1 score makes weak performance on either precision or recall harder to overlook.
• Text normalization before comparison: Describe why DSPy normalizes both strings before calculating f1_score and identify the kinds of irrelevant differences its normalize_text() method removes.
  Why it matters: Normalization keeps superficial text differences from driving a lexical comparison.
• Full test set evaluation: Describe the shift from checking lexical metrics on single examples to using DSPy’s evaluation framework and f1_score on every summary in the test set, and state the reported average.
  Why it matters: A test-set average describes the summarizer’s performance across many examples, rather than just one.

Key terms: F1 score, harmonic mean, precision, recall, f1_score, normalize_text(), normalizing text, DSPy’s evaluation framework, prog_summarizer, test set

8.5 Evaluation using embeddings

This section explains how to evaluate summaries by comparing their embeddings, how to interpret similarity scores, and why those scores can be hard to diagnose.

• Embedding-Based Summary Evaluation: Explain how to evaluate an LM-generated summary by embedding it and the ground truth summary, then comparing the resulting vectors. Describe what cosine similarity scores indicate and why this semantic test may be useful despite costing more than a lexical test.
  Why it matters: This gives you a way to score summary similarity beyond shared wording.
• Embedding Score Limitations: Explain why an embedding similarity score may not match a human judgment of summary similarity and what the section suggests trying when it does not.
  Why it matters: A similarity score is useful, but it does not reveal which features of the summaries drove the result.

Key terms: semantic similarity, embedder, embedding, ground truth summary, cosine similarity, metric function, embeddings, similarity metrics, tone, writing style

8.6 Evaluation using LLM-as-a-judge

Introduces the upcoming discussion of LLM-as-a-judge evaluation with and without reference text.

8.6.1 Metrics used with LLM-as-a-judge

The section explains how to choose and organize LLM-as-a-judge metrics for summaries, including semantic precision and recall, groundedness, and completeness.

• Focused judge questions: Explain why a judge LM may be asked specific questions about a summary rather than only giving an overall similarity score, and compare separate judge calls with one call containing multiple output fields.
  Why it matters: Focused questions let the evaluation reflect the qualities that matter for a particular project.
• Semantic precision and recall: Distinguish semantic precision and recall from lexical precision and recall, and describe the role and stated field-name limitation of DSPy’s SemanticF1.
  Why it matters: Summaries can convey the same ideas using different words.
• Groundedness versus relevance: Distinguish groundedness from precision when judging a summary, and explain why both can be useful.
  Why it matters: A summary’s claims can be supported by the document yet still miss what is most relevant.
• Completeness and comparison sources: Describe completeness and state which comparison texts can be used for precision, completeness, and groundedness.
  Why it matters: The metric determines what reference text the judge needs to see.

Key terms: LLM-as-a-judge, judge LM, output fields, semantic precision and recall, lexical precision and recall, SemanticF1, question, response, groundedness, precision

8.6.2 Metric functions using LLM-as-a-judge

The section explains how DSPy can use an LM-powered judge inside a metric function and how that judge can be optimized without leaking evaluation data.

• LM calls inside metrics: Explain that an LLM-as-a-judge metric function makes one or more LM calls to evaluate each prediction.
  Why it matters: This identifies where the judge fits into evaluation.
• Optimizable DSPy judge: Describe the judge as a DSPy program built with Predict and a simple signature, and explain that it can itself be evaluated and optimized. State that optimization can select effective demos and instructions, and that the judge LM may differ from the task and prompt LMs.
  Why it matters: A judge’s quality can be improved rather than treating its initial assessments as definitive.
• Separate judge training data: Explain why data used by the summarizers and judges is kept separate even though both programs can be optimized using training and validation data.
  Why it matters: Keeping the data separate helps avoid leakage when developing both the summarizer and its judge.

Key terms: LLM-as-a-judge, metric function, prediction, Predict, signature, judge LM, task LM, prompt LM, demos, training and validation data

8.6.3 The volume of LM calls when using LLM-as-a-judge

This section shows how LLM-as-a-judge multiplies LM calls during prompt optimization and why that cost must be weighed against potential production gains.

• Optimization LM call counts: Trace how candidate prompts, validation examples, and judge tests determine the numbers of prompt, task, and judge LM calls in the COPRO example. Describe how early stopping or minibatches can reduce evaluation calls.
  Why it matters: Counting calls reveals why a judge-based metric can make optimization expensive even when the prompt LM is called relatively rarely.
• Judge cost tradeoff: Explain why judge LM costs matter during optimization even though the judge is not used in production, and why those costs might still be worthwhile.
  Why it matters: The value of better prompts should be weighed against the potentially substantial cost of judging them.

Key terms: prompt LM, task LM, judge LM, validation set, early stopping, minibatches, optimization, evaluation, production

8.6.4 Using LLM-as-a-judge with DSPy

The section lays out a two-part DSPy workflow: calibrate an LLM judge against known scores, then use that judge to evaluate and optimize a summarizer.

• Judges as DSPy programs: Explain why an LLM judge can be evaluated and optimized, and why doing so requires up-front data and metric design.
  Why it matters: Checking the judge helps ensure that improving a task program is based on fair evaluations.
• Groundedness judge calibration: Trace how to create, evaluate, optimize, and finally test a DSPy judge that scores summaries for groundedness.
  Why it matters: A summarizer metric is only useful if the judge behind it has first been checked against known answers.
• Judge-guided summarizer optimization: Describe how the calibrated judge becomes a metric for evaluating and optimizing the summarizer, while keeping the two programs’ workflows separate.
  Why it matters: Separating judge calibration from summarizer optimization makes clear which program each evaluation is checking.

Key terms: DSPy programs, judges, metric, groundedness, ground truth score, training data, validation data, holdout test set, well calibrated, groundedness judge

8.6.5 Creating the datasets for the judge programs

This section explains how to build labeled datasets for summary-judging programs, including good and bad examples and different treatment of training versus evaluation data.

• Judge example structure: Describe how a judge example pairs a dialog and summary with a score indicating whether the summary is good.
  Why it matters: The judge needs examples of the decision it is being evaluated on, not just examples of summaries to produce.
• Good and bad summaries: Explain why judge evaluation needs good and bad summaries, and describe how this section constructs and labels those cases.
  Why it matters: Both kinds of examples reveal whether a judge distinguishes quality rather than simply approving every summary.
• Training and evaluation split: Distinguish why weak examples are added to validation and test data but not created for judge training in this setup.
  Why it matters: The evaluation sets must challenge the judge with bad cases even though those cases are not used as negative prompt demonstrations.

Key terms: judge, dialog, summary, 0.0, 1.0, ground truth summaries, true_score, good summaries, bad responses, training

8.6.6 Groundedness

This section creates a DSPy judge for how well a summary is grounded in its original document, then tests and evaluates the judge.

• Groundedness judge and document evidence: Explain that the specialized groundedness judge scores a summary against its original document by considering whether a claim in the summary is covered there.
  Why it matters: It makes clear what evidence the groundedness score is meant to reflect.
• Positive and negative judge checks: Describe how the section quickly checks the judge with a ground-truth summary and a summary from another dialog, and why that check is not a thorough evaluation.
  Why it matters: A plausible pair of outputs is a useful smoke test, not proof that the judge works across the dataset.
• Groundedness metric and error interpretation: Explain how the groundedness metric compares known true scores with the judge’s predictions, why it negates the error, and what the reported result suggests.
  Why it matters: It lets you interpret the evaluation score without mistaking a negative error score for poor groundedness.

Key terms: groundedness, judge program, original document, judge_val_data, ground truth summaries, groundedness score, groundedness metric function, true score, predicted score, higher-is-better

8.6.7 Completeness

This section introduces a DSPy judge for summary completeness and explains how to interpret and validate its initial output.

• Summary completeness judgment: Explain that a completeness judge assesses whether a summary covers key points from the document, and describe the section’s approach of creating, evaluating, and potentially optimizing a DSPy judge for that task.
  Why it matters: A completeness score is useful only if it reflects whether the summary leaves out important content.
• Judge output and validation: Explain what the example discussion and score can reveal, why the discussion should not be treated as the LM’s exact scoring process, and why one scored example is insufficient to evaluate the judge.
  Why it matters: A plausible discussion and a single score do not establish that the completeness judge is reliable.

Key terms: completeness, signature, completeness_predictor, discussion, completeness scores, full validation set, well calibrated

8.6.8 Testing both groundedness and completeness

The section shows how to combine groundedness and completeness scores to evaluate summaries, and how a custom module can make that combined evaluation optimizable.

• Combined summary evaluation: Explain why the example checks both groundedness and completeness, and describe how it produces and tests an overall score.
  Why it matters: A combined score reflects more than one aspect of summary quality, but it still needs validation.
• Optimizable combined evaluation module: Distinguish the two custom-module examples for combining groundedness and completeness, including how their submodules are optimized.
  Why it matters: The module design determines whether its judging components are optimized beforehand or together as one process.

Key terms: groundedness, completeness, simple average, validation set, custom module, submodules, f1_score, LM, optimized

8.6.9 Evaluation by Q&A

This section presents question-and-answer evaluation as a way to judge whether a summary covers expected content.

• Question-based summary evaluation: Explain how predefined questions can guide both the content of a summary and its evaluation, and how a metric function could use judge programs for those questions.
  Why it matters: It gives you a concrete way to evaluate whether a summary contains the information you expected.
• Suitable questions and limits: Describe when Q&A-based summary evaluation is useful and why it can be difficult to apply.
  Why it matters: The method is most useful when you can specify the information a summary ought to include.

Key terms: Q&A-based evaluation, predefined questions, metric function, judge programs, Q&A-based methods, content, style

8.6.10 An example of creating and using a judge

This section builds a Q&A-based summary judge, validates and optimizes it, then uses it as a metric for a baseline summarizer.

• Summary judge criteria and evidence: Explain how the judge evaluates a summary against its original document using questions about salience and named entities, while ground-truth summaries are reserved for evaluating and optimizing the judge.
  Why it matters: This setup lets you assess summaries even when no reference summary is available.
• Judge optimization and independent testing: Describe how the judge is evaluated, improved with SIMBA feedback, and checked again on previously unseen data.
  Why it matters: A judge that improves on validation data is more credible when it also performs well on unseen examples.
• Judge based summarizer metric: Explain how the optimized judge becomes a summarizer metric and how to interpret the baseline summarizer’s reported score.
  Why it matters: A high metric score is meaningful only in light of what the metric actually measures.

Key terms: Q&A, judge, ground truth summaries, salience, named entities, qa_judge_metric, judge_val_data, SIMBA, feedback, hold-out test set

8.7 Summary