Choosing between a smaller, cheaper model and a larger, stronger one requires more than intuition: compare both on the same tasks with a metric and dataset. A benchmark dataset is a curated set of tasks whose results can be compared against ground truth. section 4.2
Benchmarks make evaluation repeatable, but they are not automatically realistic. Many established benchmarks use multiple-choice questions because they are easy to collect and score, even though real users usually ask freeform questions. A benchmark can also become obsolete when models surpass it. section 4.2.4 section 12.2.4
Use benchmarks pragmatically:
- Start with established benchmarks to compare general capability, then add prompts and expected responses for specialized edge cases such as sarcasm or chess. section 4.2 section 4.2.4
- Evaluate more than task performance: segment prompts across groups and measure toxicity, polarity, hurtfulness, or other bias-related behavior. section 4.2.3
- Treat dataset curation as an operational responsibility; private data may require a separate evaluation workflow, and underrepresented languages can expose coverage gaps. section 4.2.4 section 12.2.4
Listing 4.7 Using the Evaluate library to run SQuAD
import evaluate
squad_metric = evaluate.load("squad") #1
predictions = [
{"prediction_text": "Saint Bernadette", "id": "5733be284776f41900661182"},
{"prediction_text": "Salma Hayek", "id": "56d4fa2e2ccc5a1400d833cd"},
{"prediction_text": "1000 MB", "id": "57062c2552bb89140068992c"},
] #2
references = [
{
"answers": {
"text": ["Saint Bernadette Soubirous"],
"answer_start": [515],
},
"id": "5733be284776f41900661182",
},
{
"answers": {
"text": ["Salma Hayek and Frida Giannini"],
"answer_start": [533],
},
"id": "56d4fa2e2ccc5a1400d833cd",
},
{
"answers": {"text": ["1000 MB"], "answer_start": [437]},
"id": "57062c2552bb89140068992c",
},
]
results = squad_metric.compute(
predictions=predictions, references=references
)
print(results)
# {'exact_match': 33.333333333333336, 'f1': 79.04761904761905}
Run the SQuAD metric with predictions and reference answers to see how a benchmark dataset feeds an evaluation method. section 4.2.4
