liveBook · Manning

Baseline

As explained in LLM Customization and Fine-Tuning

A reference performance level against which model changes are evaluated.

A baseline is a reference performance level for deciding whether a model change is worthwhile. Without one, a team can spend weeks adapting a model that a careful prompt already makes good enough. section 1.7 section 2.1

A prompting baseline is the score a frontier model achieves with a carefully designed prompt and no training. Measure it on a held-out set that stays out of prompts and training, then compare subsequent variants on those same examples. Also compare the score with the business requirement, the existing system and human agreement; a score alone does not establish whether the task is solved. section 2.1 section 4.9.2

Use the baseline to make the next decision:

  • Size the held-out set for the decision. Fifty examples can expose a catastrophic gap, but resolving a difference of a few percentage points typically takes 200–500. section 1.7
  • If the prompted model meets the business requirement, consider shipping it rather than training. If the gap is missing knowledge, consider RAG; if coverage is the gap, consider more examples in the prompt. section 1.5 section 4.9.2
  • Consider fine-tuning when prompting plateaus or its per-request cost becomes too high, provided there are enough high-quality, representative examples. Keep the held-out set for comparison. section 4.9.2 section 1.7

Support-ticket classification

A frontier API classifies 39 of 50 held-out tickets correctly, giving a 78% prompting baseline. That small set places the true rate somewhere from the mid-60s to the high 80s, so the result is a coarse signal. Compare it with the rules engine or human triage it would replace and with how often humans agree on the same tickets before deciding whether to adapt. section 2.1

Baselines also apply after training. For preference optimization, retain the supervised fine-tuning evaluation report: overall Token-F1, per-category scores and safety refusal rate become the reference for measuring subsequent changes. section 6.10.2

Figure 4.7 The graduation decision tree: start with prompting, add RAG when the diagnostic shows a knowledge gap, and graduate to parameter-efficient fine-tuning when prompting plateaus or per-request cost flips the economics.

Use the decision tree to move from a few-shot baseline toward retrieval, more examples or fine-tuning according to the diagnosed gap and cost. section 4.9.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Provides a measurement procedure, sample-size guidance and decision paths, including uncertainty limits for small evaluation sets.
Examples
Uses support-ticket classification and a study-guide model comparison to illustrate baseline-driven decisions.
What it assumes
Calls for a held-out evaluation set and, before fine-tuning, several hundred high-quality, domain-representative examples.
What you can do after
Evaluate whether a model change improves on a measured reference and meets a business requirement.
How it connects
Connects prompting and few-shot learning to retrieval-augmented generation, fine-tuning and preference optimization.

Also explained in

Compare how 27 books explain baseline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.