liveBook · Manning

Demonstration

As explained in LLM Customization and Fine-Tuning

A data example consisting of an input prompt and its desired completion.

A demonstration pairs an input with a desired completion. In a few-shot prompt, demonstrations show a model the task’s output pattern without updating its parameters. Similar prompt–completion pairs can also form a fine-tuning dataset, where they serve a different purpose: training rather than in-context conditioning. section 4.0 section 5.1.1

For a support classifier, each demonstration pairs a ticket with its category. A system instruction states the task; labeled examples follow in a consistent Ticket: and Category: format; the final query ends with Category: to mark where the answer belongs. section 4.2.1

Figure 4.2 Few-shot prompt anatomy: a system instruction names the task, labeled examples set the input/output schema, and a trailing “Category:” marker shows the model where its answer goes.

The prompt anatomy shows how demonstrations sit between the instruction and the new query. section 4.2.1

Eight-shot ticket routing

A team used eight hand-picked demonstrations, weighted toward high-volume categories, to classify tickets into twelve categories. After four short rounds of error-driven changes, the prompt reached 82% top-1 accuracy on 100 labeled tickets, without a training pipeline or a new checkpoint. section 4.0

Build and test a demonstration bank as an operational artifact:

  • Compare against a zero-shot baseline, then select examples across categories and keep their input and output formatting consistent. section 4.4.1 section 4.2.1
  • Group held-out failures by category and error type; add or swap examples to address the largest cluster rather than simply increasing the count. section 4.4.1 section 4.2.6
  • Measure accuracy and run-to-run disagreement, then probe brittleness by reordering examples and paraphrasing the instruction. Version the example bank separately from the prompt template. section 4.4.1 section 4.2.6

Each demonstration adds input tokens to every request. Long context windows allow many more examples, but the practical limit may be what they are worth paying for. Test the prompting baseline before committing to fine-tuning; consider retrieval when the gap is missing knowledge rather than behavior. section 4.3 section 2.1 section 4.10

A demonstration usually shows the input and final answer. For a multi-step task, a chain-of-thought example can also show reasoning before the answer, at the cost of more output tokens; a single-label classification task may not need it. section 4.2.6

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains prompt structure, example-bank iteration, validation, and the cost of adding shots.
Examples
Uses labeled support tickets and an IT question–answer dataset to illustrate prompt and training examples.
What you can do after
Evaluate a demonstration bank against a zero-shot baseline using held-out accuracy, variability, and brittleness checks.
How it connects
Connects demonstrations to in-context learning, zero-shot prompting, chain-of-thought prompting, and fine-tuning.

Also explained in

Compare how 9 books explain demonstration →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.