chapter four

4 In-context learning, few-shot, and RAG

 

This chapter covers

  • Building few-shot prompts that generalize across an input distribution
  • Scaling to many-shot when long context windows make it cost-effective
  • Validating prompts with held-out tests and run-to-run variability checks
  • Building retrieval-augmented generation (RAG) for knowledge gaps, from a 50-line minimal pipeline to chunking, reranking, hybrid retrieval, and end-to-end evaluation
  • Deciding when to graduate from prompting to parameter-efficient fine-tuning

Contoso's support team needs to route incoming tickets into twelve categories. Volume is climbing, and Maya, an engineer, opens the runbook for spinning up a fine-tuning job: data prep, GPU node, evaluation pipeline, deployment story. A colleague suggests trying few-shot first, on the same Qwen3-4B-Instruct-2507 model the team already runs. Eight hand-picked examples, weighted toward the highest-volume categories, run against a hundred labeled tickets pulled from last week. Top-1 accuracy: 82%. Two hours from idea to a number, no checkpoint to host, no training pipeline to maintain.

4.1 The case for in-context learning

4.1.1 Evolution of the approach

4.2 Few-shot prompting in practice

4.2.1 Building the prompt

4.2.2 Selection rules: Diverse, representative, no cherry-picking

4.2.3 Formatting: Consistent input/output schema

4.2.4 Ordering: The recency effect

4.2.5 Example count vs. accuracy: The empirical curve

4.2.6 A practical iteration cycle

4.3 Many-shot prompting and long-context windows

4.3.1 When many-shot wins

4.3.2 The cost trade-off

4.3.3 When many-shot does not help

4.3.4 Lost in the middle: The long-context attention U-shape

4.3.5 Building a many-shot prompt

4.4 Designing prompts for production

4.4.1 A typical validation cycle

4.4.2 The held-out test set

4.4.3 Run-to-run variability

4.4.4 LLM judging for prompt quality

4.4.5 Brittleness signals and what to do about them

4.4.6 When validation says stop tuning the prompt

4.5 Use RAG when knowledge gaps appear

4.5.1 Knowledge gaps versus behavior gaps