liveBook · Manning

Chapter 4: In-context learning, few-shot, and RAG

What you will learn in chapter 4 of LLM Customization and Fine-Tuning

The chapter opens by using a ticket-routing example to explain in-context learning and why a few-shot baseline is worth testing before fine-tuning.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter opens by using a ticket-routing example to explain in-context learning and why a few-shot baseline is worth testing before fine-tuning.

Key terms: in-context learning (ICL), prompt engineering, parameter update, few-shot baseline, top-1 accuracy, inference token costs

4.1 The case for in-context learning

The section explains why examples in a prompt can guide a pretrained model and places RAG on the same in-context learning rung.

Key terms: ICL, pre-training, prompt, RAG, in-context learning, adaptation continuum, weight-level technique

4.1.1 Evolution of the approach

This section explains how improved frontier models and longer context windows change the prompt-versus-fine-tune decision while leaving prompt validation essential.

Key terms: long-context, many-shot, few-shot, input distribution, inference compute, frontier ICL ability, reasoning models, chain-of-thought few-shot examples, held-out test sets, run-to-run variability

4.2 Few-shot prompting in practice

Introduces a support-ticket classification task and explains how few-shot examples guide—and can bias—the model without training it.

Key terms: few-shot prompting, bias, category label

4.2.1 Building the prompt

This section shows how to assemble a few-shot classification prompt and why consistent formatting helps the model follow its examples.

Key terms: few-shot prompt, system message, Ticket:, Category:

4.2.2 Selection rules: Diverse, representative, no cherry-picking

This section explains how to choose few-shot examples that reflect the range and form of production inputs without creating misleading results through cherry-picking.

Key terms: example bank, diversity, input distribution, surface variations, representativeness, production-realistic queries, test fixtures, cherry-picking, offline evaluation, stratify

4.2.3 Formatting: Consistent input/output schema

This section explains why few-shot examples and the final query should use a consistent input/output format.

Key terms: schema, field labels, separator, final query

4.2.4 Ordering: The recency effect

This section explains why the order of few-shot examples can affect accuracy and when it is worth optimizing that order.

Key terms: few-shot prompts, recency bias, most-relevant-last, category-stratified order, frontier models, run-to-run variability

4.2.5 Example count vs. accuracy: The empirical curve

This section uses an illustrative few-shot accuracy curve to show why the right example count must be measured against accuracy and cost rather than chosen by a fixed rule.

Key terms: shots, accuracy, stratified example bank, eight-shot point, input tokens, latency, practical sweet spot

4.2.6 A practical iteration cycle

This section shows how to improve a few-shot ticket classifier by diagnosing errors, replacing examples within a fixed budget, deciding when to stop, and choosing whether reasoning examples fit the task.

Key terms: few-shot example bank, coverage gap, stratified pass, edge cases, marginal cost, marginal accuracy gain, follow-up workflow, top-1 probability, logprobs, held-out set

4.3 Many-shot prompting and long-context windows

Larger context windows make many-shot prompts feasible, shifting the practical constraint from fitting examples to paying for them on every request.

Key terms: eight-shot ceiling, 32K-token context window, 1M-token window, binding constraint

4.3.1 When many-shot wins

This section explains when adding many-shot examples improves results by covering input variations, and why the gains differ across tasks.

Key terms: eight-shot accuracy, deployment threshold, coverage, capability, few-shot bank, many-shot, classification, rare categories, surface variation, structured extraction

4.3.2 The cost trade-off

This section weighs the per-request token cost of many-shot prompts against accuracy gains and explains how caching and batch pricing can change that calculation.

Key terms: many-shot prompts, shots, input tokens, example bank, input distribution, accuracy plateaus, list prices, cached input tokens, batch endpoints, fixed example bank

4.3.3 When many-shot does not help

This section explains how to recognize when many-shot prompting has stalled because of a reasoning or knowledge bottleneck rather than insufficient example coverage.

Key terms: reasoning bottleneck, multi-step inference, example bank, pre-training-knowledge bottleneck, RAG, underlying domain, input distribution, many-shot accuracy, deployment threshold, capability

4.3.4 Lost in the middle: The long-context attention U-shape

Explains why relevant examples can lose effectiveness in the middle of a long prompt and how deliberate placement improves many-shot prompting.

Key terms: lost in the middle, long-context attention U-shape, many-shot prompting, recency rule, most-relevant-last, coverage, context length

4.3.5 Building a many-shot prompt

This section explains how to assemble many-shot prompts by selecting relevant examples programmatically, and flags the latency and token-cost trade-offs.

Key terms: many-shot prompt, example bank, similarity-based selector, in-context examples, sentence-transformer embeddings, dense vector, semantic similarity, bank embeddings, vector database, token cost per request

4.4 Designing prompts for production

Explains why a prompt’s score on design examples may overstate production performance and why it needs validation before deployment.

Key terms: held-out test set, repeated runs, brittleness check, overfitting, distribution

4.4.1 A typical validation cycle

This section gives a six-step cycle for testing, improving, and checking whether a prompt is reliable enough to ship.

Key terms: held-out set, zero-shot baseline, few-shot pass, accuracy, disagreement, per-example consistency, brittleness probe, model-portability check, noise band, validation cycle

4.4.2 The held-out test set

This section explains why prompt evaluation needs unseen representative inputs and how test-set size limits the conclusions you can draw.

Key terms: held-out test set, prompt-iteration loop, generalization, smoke test, sampling error, run-to-run variability, golden test set, paired comparison, variants, tickets where they disagree

4.4.3 Run-to-run variability

This section shows why prompt validation should measure run-to-run disagreement as well as accuracy, and how to interpret both on a small ticket-classifier test set.

Key terms: run-to-run variability, temperature, brittleness, greedy decoding, disagreement rate, accuracy, validator, test set, eight-shot, two shots

4.4.4 LLM judging for prompt quality

The section explains how to judge prompt quality when exact-match metrics fall short, how to choose and check an LLM judge, and why a stronger offline judge can be affordable.

Key terms: closed-form metrics, exact-match accuracy, LLM judge, rubric, test set, prompt outputs, judge model, judge-specific bias, frontier API, production model

4.4.5 Brittleness signals and what to do about them

This section explains that prompt brittleness has multiple signals and must be measured across inputs and model upgrades rather than treated as a one-time, binary judgment.

Key terms: disagreement rate, brittleness signals, brittleness, input distribution, model upgrade

4.4.6 When validation says stop tuning the prompt

This section explains when validation should end prompt tuning and how to diagnose whether to move to RAG or a behavior-changing technique.

Key terms: held-out accuracy, noise band, run-to-run variability, capability ceiling, validation cycle, first pass, marginal cost, marginal gain, knowledge gap, behavior gap

4.5 Use RAG when knowledge gaps appear

This section explains why RAG is an in-context technique for supplying missing knowledge and offers a quick test for deciding whether retrieval will help.

Key terms: RAG, input layer, weight layer, retrieval, corpus, in-context learning, few-shot prompting, knowledge gap, behavior gap, parametric knowledge

4.5.1 Knowledge gaps versus behavior gaps

This section distinguishes missing information from incorrect output behavior so readers can choose an appropriate adaptation technique.

Key terms: knowledge gap, behavior gap, RAG, fine-tuning, inference time, format-only gaps, JSON schema, constrained decoding

4.5.2 Minimal RAG architecture

This section traces how a minimal RAG pipeline prepares documents and retrieves relevant context for an LLM.

Key terms: chunker, embedder, vector index, document-ingestion path, retriever, top-k, prompt assembly, query path

4.5.3 Vector embeddings, vector indexes, and vector databases

This section distinguishes embeddings, vector indexes, and vector databases, then explains how to choose and operate them for retrieval at different scales.

Key terms: vector embedding, cosine similarity, dot product, top-k, dimensionality, 384-dimensional all-MiniLM-L6-v2, 768 or 1024 dimensions, sentence-transformer models, encoder, generative model

4.5.4 A 50-line RAG implementation

This section describes a minimal RAG pipeline and explains why starting with a thin implementation makes retrieval quality easier to measure.

Key terms: ingest, retrieve, answer, top-k chunks, sentence-transformers, numpy array, thin pipeline, baseline, retrieval signal, reranker

4.5.5 Beyond the minimal pipeline

This section explains when and how to improve a minimal RAG pipeline through better chunking, reranking, query rewriting, hybrid retrieval, and retriever fine-tuning.

Key terms: minimal pipeline, v1 deployment, failure modes, fixed-window chunking, structure-aware chunking, semantic chunking, reranking, cross-encoder, vector similarity, first-stage retriever

4.5.6 Common RAG failure modes

Identifies four common production RAG failures and shows how to diagnose each before choosing a fix.

Key terms: Retrieved-but-not-used, retrieved content, internal knowledge, Wrong-chunk, chunking strategy, hybrid search, reranking, Hallucination on retrieval gaps, insufficient context, downstream check

4.5.7 Measuring RAG quality before debugging it

This section shows how to measure RAG retrieval separately from answer quality so debugging starts with the failing layer.

Key terms: retrieval quality, generation quality, LLM-as-judge, Precision@k, Hit@k, Hit@1, Recall@k, top-k, Precision@3, Hit@3

4.5.8 Evaluating RAG end-to-end

This section shows how to evaluate RAG retrieval, answer faithfulness, and end-to-end accuracy separately so teams can identify the bottleneck, then gives a brief rule for choosing vector search infrastructure.

Key terms: retrieval hit rate, Hit@k, retrieval step, answer faithfulness, retrieved chunks, parametric knowledge, LLM-judged, end-to-end accuracy, held-out test set, Hit@k ceiling

4.5.9 When RAG is not the answer

This section identifies when RAG is a poor fit and distinguishes problems that call for better documents, a different model-adaptation technique, or multi-step retrieval.

Key terms: RAG, source data, documents, latency budget, embedding call, index lookup, longer prompt, behavior gap, parameter-efficient, supervised

4.6 Limits of in-context approaches

This section introduces the idea that edge-case failures can signal when in-context learning has reached its limit for a task, without yet explaining those signals.

4.6.1 Diminishing returns on example count

This section explains when adding prompt examples stops paying off, how production costs can favor fine-tuning, and why prompts need retesting after model upgrades.

Key terms: shot count, first plateau, many-shot prompting, input distribution, fine-tuning, per-request inference cost, input tokens, amortized fine-tuning, latency budget, ICL prompt

4.7 Cost and decision economics

This section frames the ICL-versus-fine-tuning decision as a production cost calculation worth doing before investing in engineering.

Key terms: ICL, fine-tuning, per-request cost, request volume

4.7.1 The cost components

This section compares the per-request cost components of in-context learning and self-hosted fine-tuning to explain why their costs can break even at higher request volumes.

Key terms: ICL, input tokens, output tokens, example bank, 50-shot prompt, GPU amortization, training, hosting cost, inference compute, break-even point

4.7.2 The break-even formula

This section compares the fixed and per-request costs of ICL, fine-tuning, and RAG to explain how volume and hosting affect the economic choice.

Key terms: ICL total cost, fine-tuning total cost, break-even point, request volume, RAG, retrieval infrastructure, corpus encoding, retrieved context, knowledge-bound tasks, self-hosting

4.7.3 A worked example at four volumes

This section uses an illustrative eight-shot prompt to show how request volume, caching, and example count change the cost comparison between in-context learning and fine-tuning.

Key terms: eight-shot prompt, X per million tokens, break-even, LoRA endpoint, cacheable prefix, cached rate, uncached upper bound, payback, example bank, validation harness

4.7.4 When the math is not the only input

The section explains why cost break-even alone cannot determine whether to fine-tune and how to manage token costs as an ongoing engineering concern.

Key terms: fine-tuning, latency targets, auditable model weights, on-prem deployment, data-privacy rules, self-hosted open-weight model, token economics, tokenomics, FinOps, tokens per request

4.8 Managed platforms for prompt engineering and RAG

This section explains what managed prompt-and-RAG platforms provide and how to choose a practical mix of cloud-native services, specialty tools, and model APIs.

Key terms: prompt management, prompt observability, RAG infrastructure, frontier model access, cloud-native, hybrid pattern, specialty observability tool, audit trail, cloud-native option, hybrid search

4.8.1 What the platform layer does not solve

This section explains which evaluation and content responsibilities remain with the team when using managed platforms, then offers practical advice on prompt observability and many-shot prompting.

Key terms: held-out test set, RAG content quality, model-portability regression, evaluation primitives, many-shot, demonstration order, input distribution, most-relevant-last heuristic, prompt observability tool, prompts in production

4.9 When to graduate to parameter-efficient methods

The section explains why recognizing the measurable limits of few-shot and many-shot prompting prevents wasted prompt-engineering effort.

Key terms: few-shot, many-shot prompting, prompting toolbox, plateaued

4.9.1 Diagnostic signals that say graduate now

This section explains when a cluster of prompting-plateau signals favors parameter-efficient fine-tuning and when the apparent problem instead calls for hosting or retrieval changes.

Key terms: prompting alone, parameter-efficient methods, on-prem requirement, knowledge gap, self-hosting, retrieval

4.9.2 The decision tree

This section gives a diagnosis-based path from prompting to fine-tuning and explains how prompt-development work carries forward.

Key terms: few-shot baseline, RAG, knowledge gap, coverage gap, many-shot, per-request cost, few-shot examples, fine-tuning training set, held-out test set, prompt template

4.9.3 What changes operationally when you graduate

This section explains four operational changes to plan for when moving from prompting to fine-tuning: hosting, evaluation, versioning, and costs.

Key terms: few-shot prompt, fine-tuned model, managed inference endpoint, deployment pipeline, rollback plan, evaluation cadence, held-out test set, production input distribution, checkpoint, base model

4.10 Summary

4.11 References

This section lists references for the chapter’s discussion of prompting, retrieval-augmented generation, and cost management.

Where this is used later

Concepts explained in this chapter

← Chapter 3: What data do I need for model adaptation? Chapter 5: Parameter-Efficient Fine-Tuning: LoRA and QLoRA →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.