A baseline is a quantitative reference measured before a model is modified. It makes the effects of pruning, distillation, or specialization visible: later results can be compared with the starting point to determine whether performance improved or degraded, and by how much. section 2.2 section 2.8
Establish and use a baseline in a repeatable evaluation workflow:
- Define task-specific success metrics and validate the evaluation harness on cases with known results. Distinguish important failure types where possible. section 13.7
- Measure the original model on the task benchmark, general capability benchmarks, and inference efficiency, including latency and tokens per second on the target hardware. section 13.7
- Save the results, then repeat the same tests for each modified model using the same hardware and configuration. Set an acceptable capability loss before choosing among variants. section 13.7
- Inspect category-level results and sample counts, not just aggregate scores. Small test sets provide directional signals rather than statistically robust estimates. section 7.3.2
A task-specific baseline alone can miss damage outside the target task. General benchmarks provide a reference for detecting capability loss, including catastrophic forgetting, after specialization. Efficiency measurements supply a separate reference for deciding whether a gain justifies the optimization effort. section 7.6 section 13.7
Structured extraction baseline
Before specialization, a 4-bit SmolLM2-1.7B-Instruct model was evaluated on 40 medical extraction samples with a strict JSON prompt. Overall schema compliance was 85%. Category results exposed a weakness hidden by the aggregate: only 58.3% of responses in the irrelevant-input category matched the schema. The small sample makes these results directional. section 7.3.2
samples valid_json_pct schema_ok_pct category abbreviations 6 100.0 100.0 clean 8 100.0 100.0 implicit 7 100.0 100.0 irrelevant 12 58.3 58.3 typos 7 100.0 100.0
The category breakdown separates valid JSON from full schema compliance and shows the sample count behind each percentage. section 7.3.2
Inference reference on vLLM
An unmodified Llama-3.2-3B run on a Colab T4 provided a reference for a later pruned-model comparison. At batch size 1, vLLM reserved 5.63 GiB of KV cache and generated at 33.55 milliseconds per token. Those figures apply to that run and configuration, not to every deployment. section 12.6.2
A published model can also serve as a baseline when the goal is to beat an existing configuration rather than measure change from an untouched model. In a distillation exercise, a 14-block student with 89.5% capability retention and 10.85 perplexity is the reference to surpass on benchmarks and overall recovery. section 6.7


























