A benchmark is a standardized task or test suite used to measure and compare model performance. Establishing a baseline before changing a model makes it possible to quantify both improvements and losses after pruning, fine-tuning, or knowledge distillation. section 2.2 section 2.6
General benchmarks help track broad capabilities, but strong leaderboard results do not guarantee success on a specific production task. Pair them with a task-specific benchmark and efficiency measurements to assess whether a model is useful in its intended setting. section 1.1.2 section 13.7
Build and use a task-specific benchmark in a repeatable workflow:
- Split representative data into training and test sets with no query in both. Keep the test set untouched until measurement. section 13.7
- Choose metrics that reflect success, such as exact match, format validity, or accuracy per class. Define an acceptable tolerance when correctness is not binary. section 13.7
- Write a test harness and validate it against cases with known expected results. Check whether it distinguishes format, selection, and content errors. section 13.7
- Record the original model’s task-specific and general benchmark results alongside latency and tokens per second on the target hardware. Repeat the same tests on each modified model. section 13.7
Listing 7.7 Obtaining benchmarks
BENCHMARK_TASKS = [
"arc_easy",
"winogrande",
"hellaswag",
"lambada_openai",
"piqa"
]
BENCHMARK_LIMIT = 500
baseline_benchmarks = model_evaluation(
model, tokenizer, BENCHMARK_TASKS,
device=DEVICE, limit=BENCHMARK_LIMIT, batch_size=4
)
for task, metrics in baseline_benchmarks.items():
print(f" {task}: {metrics}")
This evaluation runs five general benchmark tasks on 500 samples each, creating a reference for measuring knowledge preservation after fine-tuning. section 7.3.2
Comparing pruning and recovery
A Gemma 3 270M baseline scored 0.59 on ARC Easy and 0.43 on LAMBADA. After depth pruning, those scores fell to 0.49 and 0.36; after knowledge distillation, they recovered to 0.54 and 0.38. The modified model remained 4.16% smaller and was reported as 10% faster. On WinoGrande, scores near the 0.50 random-chance level warrant caution: a stable score alone does not establish that reasoning was preserved. section 2.4 section 2.6
Benchmark limits can reduce evaluation time, but smaller runs may produce different numbers. Keep the evaluation settings consistent when comparing variants, and use qualitative checks alongside benchmark scores to inspect failure modes. section 2.2 section 2.4





























