Chapter 7: Prompt engineering: Becoming an LLM whisperer

Introduction

The introduction explains why prompting lets one general-purpose LLM serve many tasks and previews the chapter’s approach to prompt engineering.

• Task reuse through prompting: Explain why an LLM can be used for different tasks without training a separate model for each one, and how prompts direct the same model toward a particular use case.
  Why it matters: Prompting is the chapter’s starting point for using an existing model across applications.
• Prompt engineering chapter scope: Describe the chapter’s promised progression from what prompts are and how to make them, through prompt engineering and its tooling, to advanced techniques for difficult questions.
  Why it matters: The roadmap shows that the chapter treats prompt engineering as more than writing a single prompt.

Key terms: prompt, prompt engineering, in-context learning, general-purpose tool, prompt engineering tooling, advanced prompting techniques

7.1 Prompting your model

This section defines prompting as crafting model input for a desired output and introduces prompt engineering as a way to make LLM application behavior more consistent without retraining.

• Prompt as purposeful input: Distinguish the basic meaning of a prompt—the input to a language model—from prompting as deliberately crafting that input to get a desired output.
  Why it matters: This distinction helps you recognize when an application is shaping a model’s response rather than merely passing along a user’s message.
• Guidance without model retraining: Explain why a carefully crafted prompt can help bring out information that is not the model’s most probable response, and why this is useful in applications without expensive retraining.
  Why it matters: Prompting offers a way to steer an existing model toward useful answers without paying to retrain it.
• Prompt engineering workflow: Describe prompt engineering as designing, templating, and refining prompts, then implementing what works in code to produce meaningful, consistent user experiences.
  Why it matters: Application teams need repeatable prompt behavior, not just a single good-looking response.

Key terms: prompt, prompting, probable responses, parameters, retraining, prompt engineering, designing, templating, refining

7.1.1 Few-shot prompting

This section shows how a few examples can steer an LLM’s task and output format, while explaining the limits imposed by token budgets and cost.

• Few-shot example patterns: Explain that few-shot prompting gives an LLM several examples of the desired behavior so it can follow their pattern. Describe how labeled reviews can lead it to classify a new review as positive or negative without an explicit instruction.
  Why it matters: Examples can communicate both the task and the expected response more directly than instructions alone.
• Output guidance and examples: Describe how instructions can specify allowed answers and restrict the response to one option, while examples can demonstrate a required structure such as JSON or XML. Explain that merely asking for JSON may still produce subtle errors, such as single quotes instead of double quotes.
  Why it matters: A response can convey the right idea yet still fail when an application expects a precise label or data structure.
• Example length tradeoffs: Explain that few-shot examples consume context-window tokens and can be especially long for code. Describe the tradeoff between adding examples and leaving room for user input, as well as the per-token cost that motivates one-shot prompting.
  Why it matters: A useful example may not be worth its space or price in a production prompt.
• Prompt wording conventions: Explain the section's suggestion that polite wording may help because useful, well-structured human conversations in training data often follow politeness conventions. Distinguish this from a guarantee: the section says other styles can also work but may be less consistent.
  Why it matters: Prompt wording can draw on patterns the model encountered in training, not just literal commands.

Key terms: few-shot prompting, examples, sentiment analysis, instructions, JSON, XML, structured outputs, token limits, context limits, per token

7.1.2 One-shot prompting

The section explains how one-shot prompting uses a single example to guide an LLM, why it can work, and where it can fail.

• One-shot prompt structure: Describe one-shot prompting as giving an LLM one clear instruction or example to guide its response to a new task, rather than supplying several examples. Explain how the section’s prompt pairs a worked word problem with a new word problem.
  Why it matters: Recognizing what the single example is meant to teach helps you judge whether a prompt is actually one-shot.
• Reasons one example can work: Explain that broad pretraining gives LLMs patterns and concepts they can apply to a new task, even when the exact task was not in training. Relate this to the section’s claim that the first prompting example does the heaviest lifting and later examples offer diminishing returns.
  Why it matters: This explains why a single carefully chosen example can be useful despite seeming insufficient.
• One-shot benefits and limits: Describe why one-shot prompts are faster and easier to craft and experiment with than prompts needing more examples. Explain that a lone example may bias the answer, while results still vary with task complexity, specificity, and model scale.
  Why it matters: A single example saves effort, but its choice and the task’s difficulty still affect the result.

Key terms: one-shot prompting, one-shot learning, instruction, example, pretraining, generalize, diminishing returns, bias, model size, task complexity

7.1.3 Zero-shot prompting

This section explains how zero-shot prompts can steer an LLM without examples, especially through reasoning cues, while emphasizing tradeoffs and model-specific results.

• Zero-shot prompts without examples: Explain that zero-shot prompting aims to get an expected result without supplying examples, and describe how a simple Q: [user’s prompt] A: template can prime a model to answer a question.
  Why it matters: It distinguishes zero-shot prompting from the example-based approaches discussed earlier.
• Step-by-step reasoning cues: Explain how asking a model to “think step by step” can elicit chain-of-thought reasoning without worked examples and improve answers to multi-step problems such as math. Describe the downside of longer responses that may continue after the answer.
  Why it matters: A short instruction can improve reasoning, but its effect on response length matters in an application.
• Model-specific prompt experimentation: Describe why the section recommends testing zero-shot wording rather than expecting one universally best prompt: evaluated variations can outperform “think step by step” for a particular model, and even human-style appeals can affect outputs. State that such effects may depend on the model and its training data.
  Why it matters: A prompt that works well for one model or task may not transfer to another.

Key terms: zero-shot prompting, Q: [User’s Prompt] A:, Chain of Thought (CoT), think step by step, stopping criteria, thread of thought, GPT-4

7.2 Prompt engineering basics

This section distinguishes prompt engineering from casual prompting and explains when to try it before changing a model.

• Reliable prompts for varied users: Explain why getting a useful answer through trial and error is different from engineering prompts for an application: the application must produce answers in the expected format for users with different prompting skills while resisting attempts to derail it.
  Why it matters: An application should not depend on each user knowing how to coax the right answer from the model.
• Prompting before model changes: Describe the section's recommendation to try prompt engineering before finetuning or using a LoRA, while recognizing that prompt engineering alone has limits, even with RAG, and that further finetuning may be needed.
  Why it matters: Recognizing when prompts have reached their limit can prevent weeks of unproductive work.

Key terms: prompt engineering, prompting, format, railings, finetuning, PEFT, LoRA, retrieval-augmented generation (RAG)

7.2.1 Anatomy of a prompt

This section breaks an application prompt into input, instruction, context, and system prompt, using an internal HR chatbot to show what each contributes.

• User input and instructions: Distinguish the user's free-form input from the instruction template that wraps it and directs how the model should answer, using the HR question as an example.
  Why it matters: Separating what the user wrote from the application's instructions makes prompts easier to design and evaluate.
• Context for grounded answers: Explain how context supplies information the model needs to answer appropriately, including retrieved company documents in the HR example and other dynamic or static information.
  Why it matters: A model cannot reliably answer a company-specific question unless the relevant information is supplied to it.
• System prompt behavior and safety: Describe how a system prompt is included on every request to establish consistent behavior or style, and explain the section's cautions about prompt injection and sensitive information.
  Why it matters: The system prompt helps maintain a consistent experience, but treating it as a place for secrets creates a security risk.

Key terms: input, instruction, template, Q&A bot template, context, RAG, semantic search, few-shot, one-shot prompting, system prompt

7.2.2 Prompting hyperparameters

This section explains how generation settings beyond the prompt affect an LLM’s predictability, diversity, quality, and latency.

• Hyperparameter tuning and availability: Explain that generation hyperparameters are set alongside a prompt to shape responses, and that an LLM API endpoint must accept a parameter before it can be used.
  Why it matters: A useful setting has no effect if the endpoint does not support it.
• Temperature and token probabilities: Explain how temperature changes next-token selection: logits become probabilities through softmax, higher temperature flattens their distribution, and lower temperature favors the highest-scoring tokens. Describe why a setting of zero uses argmax for repeatable output rather than division by zero.
  Why it matters: Temperature is a principal way to trade predictable answers for more varied ones.
• Beam search size tradeoff: Describe beam search as exploring promising paths of possible next tokens, and explain that more beams can improve response flow and quality at the cost of latency.
  Why it matters: Beam count offers a quality-versus-speed choice when generating a response.
• Top K and Top P filtering: Distinguish top K, which restricts the next token to a fixed number of most-probable candidates, from top P, which selects candidates according to a probability threshold. Explain how tighter filtering can make output more rigid while looser filtering allows more variety and potentially incoherent text.
  Why it matters: These settings control which candidate tokens remain eligible for generation.
• Frequency and presence penalties: Distinguish frequency penalties, which discourage recently reused words, from presence penalties, which penalize a repeated token the same whether it appears twice or many times. Explain how the former promotes varied wording and the latter aims to encourage new topics or ideas.
  Why it matters: The two penalties address different kinds of repetition in generated text.

Key terms: prompt hyperparameter tuning, LLM API endpoint, temperature, logits, softmax, argmax, beam search, number of beams, latency, top K

7.2.3 Scrounging the training data

This section shows how inspecting a model’s training data can reveal effective wording, especially when prompt engineering is paired with finetuning.

• Training data vocabulary: Explain why effective wording may depend on the terms used in a model’s training data. Describe how the author searched LAION captions after several DreamBooth base classes produced poor images, then succeeded with the caption phrase “Asian beauty.”
  Why it matters: Looking at the data can reveal useful wording that guessing alone misses.
• Finetuning and prompting together: Explain why the section recommends using finetuning and prompt engineering together rather than treating them only as alternatives. Describe how phrases chosen during finetuning can seed the model and the same wording in a later prompt can help recall that information.
  Why it matters: Coordinating the wording used in finetuning and prompts can provide two opportunities to improve results.

Key terms: LAION, DreamBooth, base class, captions, Asian beauty, finetuning, prompt engineering, seed, recall

7.3 Tooling for structured outputs, workflows, and more

This introduction explains why applications that do more than wrap an LLM need prompt-engineering tools and sets up a look at prominent tools.

• Tooling for richer applications: Explain that applications doing more than simply wrapping an LLM may use prompt engineering to add function or personality, and that tools can help put those prompts to work when building the application.
  Why it matters: This connects the prompt-engineering basics to the tools introduced next.

Key terms: prompt engineering, tools, function, personality

7.3.1 LangChain

This section introduces LangChain chains, shows how LCEL connects prompt, model, and parsing steps, and describes reusable chains and supporting tools.

• LCEL chain composition: Trace the example LangChain chain from a prompt template through an LLM to a parser that returns a string. Explain that LCEL combines basic components into chains, which can also include asynchronous, non-linear steps like a data pipeline or DAG.
  Why it matters: Thinking of a chain as a pipeline makes its steps easier to build and trace.
• Reusable chains and tradeoffs: Explain why the simple example is arguably overkill for an f-string prompt, while existing community chains are more likely to be useful for substantial tasks. Give examples such as RetrievalQA for RAG or chains for SQL, APIs, or synthetic data.
  Why it matters: This distinction helps you decide when a chain earns its added structure.
• LangChain hosting and tracing tools: Distinguish LangServe, which hosts a chain as an API, from LangSmith, which logs and traces an invocation so you can inspect results at each link.
  Why it matters: The two tools address different needs once a chain has been built.

Key terms: LangChain Expression Language, LCEL, chain, prompt template, parser, DAG, f-string prompt, RetrievalQA chain, RAG, LangServe

7.3.2 Guidance

This section introduces Guidance as a way to constrain LLM outputs and build reusable response rules, while distinguishing reliable output structure from accurate predictions.

• Programmatic response constraints: Explain how Guidance replaces prompt-and-pray with programmatic control over generated responses, using tools such as constrained selections, custom stopping tokens, and templating.
  Why it matters: Constraining the response space can make an LLM application’s output more predictable.
• Reusable Guidance grammars: Describe a Guidance grammar as composable, reusable language rules the model must follow, created by defining a function with the @guidance decorator; relate this to the parts-of-speech application.
  Why it matters: Reusable grammars turn output constraints into building blocks for applications.
• Output reliability and accuracy: Distinguish getting the expected output form from getting good parts-of-speech predictions, and describe the section’s suggested ways to improve prediction quality.
  Why it matters: A controlled response format does not fix weaknesses in the underlying model.

Key terms: Guidance, prompt-and-pray, response space, stopping tokens, templating, guidance HF wrapper, grammars, @guidance decorator, parts-of-speech application, parts of speech prediction

7.3.3 DSPy

This section introduces DSPy as a way to program and optimize LLM prompting through defined tasks, modules, examples, and validation rather than hand-written prompt strings.

• DSPy prompting approach: Explain how DSPy differs from tools for writing prompts: it lets you define an LLM workflow programmatically and uses examples and validation to find effective prompts. Describe its emphasis on augmenting LLMs with tools such as retrieval.
  Why it matters: This distinction helps you decide whether to handcraft prompts or treat prompting more like a software engineering task.
• DSPy program compilation workflow: Describe the steps given for using DSPy: specify task signatures with input and output fields, choose a predictor or generation style, define a module, then compile it using examples and validation. Relate the examples, validation metric, and teleprompter to a training set, loss function, and optimizer.
  Why it matters: These steps show what you must define before DSPy can optimize a prompting workflow.

Key terms: DSPy, tool augmentation, retrieval, program prompting, signature, predictor, module, compile, validation metric, teleprompter

7.3.4 Other tooling is available but …

This section surveys other prompt-tooling projects and urges caution when choosing among them because the ecosystem is young and some projects have become inactive.

• Tooling longevity and selection: Explain why the authors advise caution when choosing prompt-engineering tools: some projects attracted early communities but have since seen little activity, and the ecosystem is still young. Describe the authors’ suggestion to contribute to a tool you find useful.
  Why it matters: A promising tool is more useful when its maintenance and community can support continued use.
• Alternative tooling landscape: Describe how the section positions other tools: MiniChain and AutoChain are lightweight alternatives to LangChain, Promptify is a full-feature alternative, and Outlines is similar to Guidance. Note that the authors name other tools worth exploring without trying to provide an exhaustive list.
  Why it matters: These examples show that alternatives differ in scope, even though the section’s main advice is to choose carefully.

Key terms: sponsorship, open source projects, GitHub contribution, MiniChain, AutoChain, Promptify, Outlines, LangChain, Guidance

7.4 Advanced prompt engineering techniques

This section introduces tool access as a way to address missing or changing context that prompts alone cannot provide.

• Tools for missing context: Explain why a well-designed prompt may still leave a model without needed context, and contrast supplying that context through RAG with giving the model access to tools.
  Why it matters: It explains why prompt engineering may need capabilities beyond better wording or retrieved context.

Key terms: training date, RAG, tools

7.4.1 Giving LLMs tools

This section explains how LLMs can call external tools, how prompting can add tools without finetuning, and what makes tool use difficult in practice.

• Purpose of external tools: Explain how access to tools such as search, calculators, clocks, and weather services lets an LLM obtain current information or perform tasks better handled outside the model.
  Why it matters: Tool access can make an assistant useful for tasks it cannot reliably handle from its own generated text.
• Tool call execution flow: Trace how Toolformer-style API tags signal a call that an interpreter executes during inference, and how Guidance can instead detect a prompted tool call, pause generation, run the tool, and resume generation without finetuning for each new tool.
  Why it matters: A tool call works only when something outside the LLM recognizes and executes it.
• Tool design and safety limits: Describe the practical requirements and limits of LLM tool use: prompts must explain where and how to use tools, inputs and errors need safeguards, tool access can create unintended actions, and tool descriptions consume tokens while models may ignore or choose the wrong tool.
  Why it matters: A working tool is not enough if the model calls it incorrectly, unsafely, or not at all.
• Plugin ecosystem prospects: Describe OpenAI's Plugins as an attempt to let third parties expose APIs to ChatGPT through hosted OpenAPI and ai-plugin.json files, and distinguish that attempt from the section's hoped-for, not-yet-available model-agnostic tool marketplace.
  Why it matters: The plugin example shows both the appeal of reusable third-party tools and the gap between that idea and the available ecosystem.

Key terms: tools, API calls, Toolformer, <API></API>, inference, Guidance, finetune, string interpolation, tool inputs, error handling

7.4.2 ReAct

This section explains ReAct’s thought–action–observation loop and illustrates how an LLM can use tools through LangChain to answer questions that require multiple steps.

• ReAct reasoning and action loop: Explain how ReAct prompts an LLM to reason about a question, choose an action, observe its result, and use that observation to decide what to do next or when to finish. Distinguish ReAct from prompting that only produces text by noting that the model can take actions such as searching for information.
  Why it matters: The loop explains how an LLM can gather information and revise its next step rather than answer immediately.
• LangChain ReAct tool example: Describe the section’s LangChain example as using ReAct with an OpenAI model and tools including google-serper for search and llm-math for calculation. Explain that combining tools with prompting helps address more in-depth questions than wording a prompt alone.
  Why it matters: The example connects the ReAct pattern to tools an application can give an LLM.

Key terms: Reasoning and Acting (ReAct), Thought, Action, Observation, Finish, few-shot, LangChain, OpenAI model, google-serper, serper.dev

7.5 Summary

This summary recaps how examples and prompt structure shape LLM requests, then distinguishes tools for building workflows, controlling output, and enabling tool use and reasoning.