Evaluate is a Hugging Face framework for building and applying machine-learning evaluation methods and metrics. It separates datasets from the library, storing them in public or private Hugging Face Hub spaces, which makes private benchmarks easier to manage. section 4.2.4
Use Evaluate when you need to:
- Load standard benchmarks and custom metrics, including SQuAD’s exact-match and F1 measures. section 4.2.4
- Create a custom metric and the dataset used by a new benchmark in a metric space. section 4.2.4
- Treat metric discovery cautiously because Hub uploads vary in quality, curation, and cleanliness. section 4.2.4
- Extend evaluation beyond performance to consider bias and potential harm. section 4.6
Listing 4.7 Using the Evaluate library to run SQuAD
import evaluate
squad_metric = evaluate.load("squad") #1
predictions = [
{"prediction_text": "Saint Bernadette", "id": "5733be284776f41900661182"},
{"prediction_text": "Salma Hayek", "id": "56d4fa2e2ccc5a1400d833cd"},
{"prediction_text": "1000 MB", "id": "57062c2552bb89140068992c"},
] #2
references = [
{
"answers": {
"text": ["Saint Bernadette Soubirous"],
"answer_start": [515],
},
"id": "5733be284776f41900661182",
},
{
"answers": {
"text": ["Salma Hayek and Frida Giannini"],
"answer_start": [533],
},
"id": "56d4fa2e2ccc5a1400d833cd",
},
{
"answers": {"text": ["1000 MB"], "answer_start": [437]},
"id": "57062c2552bb89140068992c",
},
]
results = squad_metric.compute(
predictions=predictions, references=references
)
print(results)
# {'exact_match': 33.333333333333336, 'f1': 79.04761904761905}
A compact Python example loads the SQuAD metric, supplies prediction and reference records, and runs question-answering evaluation. section 4.2.4
