Recall measures how much of the relevant or actual positive content a system correctly identifies. In text evaluation, it asks how much of a human reference summary a generated summary captures. section 12.2.2 section 13.5.3
Use recall to assess coverage, while checking the trade-offs around missed and incorrectly identified cases.
- Pair recall with precision and F1 to balance coverage against the relevance of detected content. section 13.5.3 section 9.3.2
- Monitor false negatives and false positives because a single recall value does not fully describe filter reliability or user impact. section 13.5.3
- Treat recall as one signal among semantic, qualitative, operational, bias, and compliance checks. section 12.2.2 section 13.5.3
Reference coverage and content filters
For summarization, inspect how much reference-summary content appears in the generated summary. For a content filter, measure how comprehensively it identifies actual harmful cases, then review false negatives, user feedback, expert analysis, and A/B-tested settings. section 12.2.2 section 13.5.3
Table 9.1 Fine-tuning evaluation criteria
|
Evaluation criteria
|
Description
|
|---|---|
|
Relevance
|
Gauges how well the model’s responses or outputs align with the context and intent of the input. This is especially crucial in applications such as chatbots, where providing contextually appropriate responses is key to user satisfaction. Relevance is often assessed by examining whether the model can stay on topic and provide information or responses directly applicable to the queries or tasks.
|
|
Coherence
|
Refers to the logical consistency of the model’s outputs. A fine-tuned model should generate contextually relevant, logically sound, and coherent text. This means the responses should follow a logical structure and narrative flow, making sense in the conversation or text context. Coherence is vital for maintaining user engagement and ensuring the model’s outputs are understandable and meaningful.
|
|
Accuracy
|
This particularly comes into play when the model is used for tasks involving factual information, such as educational tools, informational bots, or any application where providing correct information is critical. Accuracy is measured by how well the model’s responses align with factual correctness and objective truth.
|
|
Language fluency
|
Pertains to the grammatical and syntactical correctness of the model’s outputs. Even if a model is highly relevant, coherent, and accurate, poor language fluency can significantly detract from the user’s experience. This includes proper grammar, punctuation, and style, ensuring the text generated is correct and reads naturally to the end user.
|
Use the evaluation-criteria table to place recall alongside relevance, coherence, and other complementary measures when assessing adapted model outputs. section 9.3.2
