liveBook · Manning

Many-shot prompting

As explained in LLM Customization and Fine-Tuning

Prompting with a relatively large number of examples to demonstrate the desired behavior.

Many-shot prompting puts a relatively large bank of demonstrations in the prompt so a model can follow their pattern without updating its parameters. A typical range is 50–200 examples, made practical by long context windows. The extra coverage can help when a few examples miss important variations, but every request pays for the demonstrations again. section 4.0 section 4.1.1 section 4.3.2

Start with a few-shot baseline. Scale the example bank when errors point to insufficient coverage, especially across rare classes, varied input formats, or boundaries between similar categories. Classification with many categories is a strong candidate; summarization often gains less because a small set can establish the desired style. Gains on classification should not be assumed to transfer to reasoning or generative tasks. section 4.9.2 section 4.3.1 section 4.1.1

Support ticket routing

For a ticket taxonomy expanded from 12 to 50 classes, eight examples leave rare classes uncovered. Increasing the bank from eight to 50 examples raises reported top-1 accuracy from 82% to 89% by covering subcategories the smaller bank misses. section 4.3.1

Build and evaluate a many-shot prompt:

  • Store examples in a JSONL bank with inputs, outputs, and optional categories. Select a subset for each query rather than assembling a large prompt by hand. section 4.3.5
  • Use semantic similarity to select relevant demonstrations. Precompute bank embeddings or store them in a vector database; encode the query at request time. section 4.3.5
  • Compare shot counts on held-out queries and measure run-to-run variability. Use larger test sets for smaller differences: a few hundred examples can support reporting small gaps. section 4.1.1 section 4.10

More examples increase token cost roughly linearly, while accuracy can plateau. For example, 100 demonstrations at 200 tokens each add about 20,000 input tokens before the query. A fixed example bank can benefit from prefix caching; query-specific similarity selection reduces the shared prefix and may sacrifice much of that discount. Check current provider pricing and batch options. section 4.3.2

Table 4.2 Cost framing at different shot counts (illustrative)
Shots
Prompt tokens (illustrative)
Inference cost rank
Accuracy plateau range
1
~250
Low
60 to 70%
8
~1,800
Low
78 to 84%
50
~10,200
Medium
86 to 90%
100
~20,200
High
88 to 92%
200
~40,200
very high
89 to 93%

Compare illustrative prompt sizes, cost ranks, and accuracy ranges across shot counts; the ranges are not guarantees for a particular task. section 4.3.2

Choose retrieval-augmented generation when the problem is missing knowledge rather than missing example coverage. Consider fine-tuning when many-shot accuracy plateaus or repeated inference costs outweigh the cost of training and operating a tuned model. Keep the held-out set to compare alternatives on the same queries. section 4.9.2 section 4.10

How LLM Customization and Fine-Tuning covers it

How deep it goes
Works through example-bank selection, embedding-based assembly, validation, and inference-cost tradeoffs.
Examples
Uses support-ticket classification, varied invoice extraction, and English-to-Bemba translation to illustrate coverage gains.
What you can do after
Evaluate many-shot prompting against few-shot and fine-tuning using held-out accuracy, variability, and per-request cost.
How it connects
Connects in-context learning to few-shot prompting, retrieval-augmented generation, and fine-tuning.

Also explained in

Compare how 3 books explain many-shot prompting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.