liveBook · Manning

Baseline

As explained in Rearchitecting LLMs

A quantitative baseline is an objective reference measurement established before interventions to evaluate their effects.

A baseline is a quantitative reference measured before a model is modified. It makes the effects of pruning, distillation, or specialization visible: later results can be compared with the starting point to determine whether performance improved or degraded, and by how much. section 2.2 section 2.8

Establish and use a baseline in a repeatable evaluation workflow:

  • Define task-specific success metrics and validate the evaluation harness on cases with known results. Distinguish important failure types where possible. section 13.7
  • Measure the original model on the task benchmark, general capability benchmarks, and inference efficiency, including latency and tokens per second on the target hardware. section 13.7
  • Save the results, then repeat the same tests for each modified model using the same hardware and configuration. Set an acceptable capability loss before choosing among variants. section 13.7
  • Inspect category-level results and sample counts, not just aggregate scores. Small test sets provide directional signals rather than statistically robust estimates. section 7.3.2

A task-specific baseline alone can miss damage outside the target task. General benchmarks provide a reference for detecting capability loss, including catastrophic forgetting, after specialization. Efficiency measurements supply a separate reference for deciding whether a gain justifies the optimization effort. section 7.6 section 13.7

Structured extraction baseline

Before specialization, a 4-bit SmolLM2-1.7B-Instruct model was evaluated on 40 medical extraction samples with a strict JSON prompt. Overall schema compliance was 85%. Category results exposed a weakness hidden by the aggregate: only 58.3% of responses in the irrelevant-input category matched the schema. The small sample makes these results directional. section 7.3.2

                     samples  valid_json_pct  schema_ok_pct
category                                             
abbreviations        6            100.0             100.0
clean                8            100.0             100.0
implicit             7            100.0             100.0
irrelevant           12            58.3              58.3
typos                7            100.0             100.0

The category breakdown separates valid JSON from full schema compliance and shows the sample count behind each percentage. section 7.3.2

Inference reference on vLLM

An unmodified Llama-3.2-3B run on a Colab T4 provided a reference for a later pruned-model comparison. At batch size 1, vLLM reserved 5.63 GiB of KV cache and generated at 33.55 milliseconds per token. Those figures apply to that run and configuration, not to every deployment. section 12.6.2

A published model can also serve as a baseline when the goal is to beat an existing configuration rather than measure change from an untouched model. In a distillation exercise, a 14-block student with 89.5% capability retention and 10.85 perplexity is the reference to surpass on benchmarks and overall recovery. section 6.7

How Rearchitecting LLMs covers it

How deep it goes
Provides a procedural workflow for selecting metrics, validating a harness, recording measurements, and repeating tests under matching conditions.
Examples
Uses JSON schema compliance, a distilled student challenge, and a vLLM cache and latency probe as reference measurements.
What you can do after
Practice establishing task, general-capability, and efficiency baselines and comparing modified models with the same test battery.
How it connects
Links depth pruning and knowledge distillation to performance comparison, and specialization to catastrophic forgetting.

Also explained in

Compare how 27 books explain baseline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.