8 Summarization and more effective metric functions
This chapter covers
- Using DSPy for text summarization
- Creating more sophisticated metric functions
- Lexical and semantic similarity
- LLM as a judge evaluation
To evaluate or optimize any DSPy programs we create, we must have metric functions that can reliably evaluate the output they produce. So far we’ve worked primarily with classification problems, which allow for fairly straight-forward metric functions. Many other tasks required when working with LMs also require simple metric functions similar to these. For example, with entailment, as we saw in chapter 2, we test if one piece of text follows logically from another. If the LM returns only a boolean value indicating if there's logical entailment or not, the metric function can be very simple, just indicating if the LM’s response is correct or not. This is often true as well for question answering, at least when working strictly with questions that have short answers, for example, “What is the capital of France?” But, for many other LM-based tasks that we'll likely need to handle, much more complicated metric functions will be necessary.