8 Summarization and more effective metric functions
This chapter covers
- Using DSPy for text summarization
- Creating more sophisticated metric functions
- Lexical and semantic similarity
- LLM-as-a-judge evaluation
To evaluate or optimize any DSPy programs we create, we must have metric functions that can reliably evaluate their output. So far, we’ve worked primarily with classification problems, which allow for fairly straightforward metric functions. Many other tasks required when working with language models (LMs) also use simple metric functions similar to these. For example, with entailment, as we saw in chapter 2, we test whether one piece of text follows logically from another. If the LM returns only a Boolean value indicating whether there’s logical entailment, the metric function can be very simple, just indicating whether the LM’s response is correct. This is often true for question answering as well, at least when working strictly with questions that have short answers, for example, “What is the capital of France?” But for many other LM-based tasks that we’ll likely need to handle, much more complicated metric functions will be necessary.