Chapter 7: Custom modules

Introduction

The introduction explains why custom DSPy modules are useful for implementing prompting techniques and building LM-based applications.

• Purpose of custom modules: Explain how custom modules extend DSPy beyond built-in modules by letting developers implement prompting techniques and build applications such as agents or RAG systems.
  Why it matters: Custom modules let you adapt DSPy to an application rather than relying only on its built-in modules.

Key terms: custom modules, built-in modules, prompting techniques, agents, retrieval-augmented generation (RAG)

7.1 Types of DSPy modules

This section distinguishes single-call prompting modules from multistep workflows and explains how DSPy’s modular design makes custom modules reusable and optimizable.

• Two types of modules: Distinguish a module that makes one LM call to encode a prompting technique from a workflow that combines multiple LM calls with logic and possibly tools. Describe how a workflow can be specific to an application or reusable as a general prompting technique.
  Why it matters: This distinction helps you recognize when a prompting technique needs only one call and when an application needs a workflow.
• Submodule composition and reuse: Explain how built-in and custom modules are composed from submodules, with Predict as their ultimate basis, and why stacking existing modules reduces repeated code.
  Why it matters: Composition lets you build on existing DSPy modules rather than reimplementing their behavior.
• High-level module development: Describe why custom-module code can stay high level: adapters handle direct LM interaction, while lower levels of DSPy normally handle retries and backoff.
  Why it matters: These built-in responsibilities let you concentrate on the behavior your module adds.
• Custom module optimization: Explain that custom modules can be evaluated and optimized like built-in modules, including across a complete workflow, while optimizers improve prompts rather than design the workflow’s logic and sequence of LM calls.
  Why it matters: You can optimize a custom workflow without mistaking prompt optimization for workflow design.

Key terms: DSPy modules, LM call, prompting technique, workflow, submodules, Predict, ChainOfThought, modularity, adapters, retries

7.2 Coding a custom module

This section explains the required structure of a DSPy custom module, how its forward method coordinates work, and when the module itself needs a signature.

• Custom module class structure: Describe the required shape of a custom module and the roles of its main methods.
  Why it matters: This structure lets you recognize or write a custom module without confusing its required methods with the choices you make inside them.
• Forward method data flow: Trace how a custom module supplies inputs to submodules and processes their results, including a sequence of submodule calls.
  Why it matters: Following the data through forward() makes multi-step modules easier to design and explain.
• Custom module signature choices: Distinguish a fixed-purpose custom module that needs no module-level signature from a general-purpose one that accepts a signature.
  Why it matters: You can choose a fixed interface for a specific task or accept a signature when the same module must work with different fields.

Key terms: dspy.Module, __init__(), forward(), Prediction, submodules, tools, inputs, outputs, signature, input fields

7.3 DSPy’s built-in modules

Introduces the four built-in DSPy modules covered in the following subsections.

7.3.1 ChainOfThought

Explains how ChainOfThought adds a reasoning output to a signature while making just one Predict call, and what that design shows about building custom DSPy modules.

• ChainOfThought reasoning output: Explain how ChainOfThought requests reasoning by changing a signature and why the position of that output matters.
  Why it matters: This shows how a module can change an LM’s requested output without adding another LM call.
• Initialization before Predict creation: Trace why ChainOfThought changes the signature in __init__() before creating its Predict submodule, and describe the stated benefit of leaving forward() largely unchanged.
  Why it matters: The order of initialization determines which signature Predict uses and can keep asynchronous support simpler.
• Signature and rationale configuration: Describe how the simplified module prepares a string signature for modification and how the rationale output field can be configured.
  Why it matters: These are the concrete setup steps behind the section’s simpler custom-module pattern.
• Module docstring value: Explain why a custom module can benefit from a clear docstring even though one is not required.
  Why it matters: A clear description may help an optimizer interpret what the module is meant to do.

Key terms: ChainOfThought, Predict, rationale field, OutputField, signature, __init__(), forward(), aforward(), Predict submodule, ensure_signature()

7.3.2 BestOfN

This section explains how BestOfN runs one submodule repeatedly, scores its predictions, and returns the best one.

• Repeated prediction and selection: Trace how BestOfN makes up to N calls to one submodule with different LM temperatures, keeps the highest-scoring prediction, and may stop early when a score exceeds a threshold.
  Why it matters: This explains both why BestOfN makes multiple LM calls and why it still returns only one response.
• Inference-time reward scoring: Explain what the reward function evaluates, why it cannot rely on ground-truth outputs, and why its scores must be higher-is-better.
  Why it matters: A usable reward function must rank live predictions without access to a known correct answer.
• Flexible submodule input forwarding: Describe how BestOfN accepts an existing submodule and forwards its inputs through **kwargs without knowing that submodule’s signature.
  Why it matters: This shows how the same BestOfN wrapper can work with different supplied modules and inputs.

Key terms: BestOfN, submodule, N, temperature, best_pred, threshold, reward_fn, reward function, inference time, ground truth output fields

7.3.3 MultiChainComparison

Explains how MultiChainComparison combines earlier predictions and their reasoning into one final prediction, using a single Predict submodule.

• Comparison of prior predictions: Explain that MultiChainComparison takes predictions and reasoning from two or more earlier program calls and produces one final prediction, which can be new rather than selected from the inputs.
  Why it matters: This distinction clarifies when comparing several answers offers more than simply picking a winner.
• Single call comparison flow: Trace how the module prepares its Predict submodule and uses it: __init__() builds a signature for the original question and collected predictions, while forward() formats those inputs, makes one LM call, and returns the output.
  Why it matters: The flow shows how multiple earlier results become the context for one final LM decision.
• Reasoning fields and output selection: Describe which information the example takes from earlier outputs: it treats the last signature output field as the answer to improve and accepts either a reasoning or rationale field when collecting explanations.
  Why it matters: These field choices explain what the example compares and why its output can be compared again.

Key terms: MultiChainComparison, prediction, BestOfN, Predict, __init__(), forward(), signature, completions, kwargs, last output field

7.3.4 Refine

This section explains how DSPy Refine scores a submodule’s responses, generates advice after unsuccessful attempts, and uses that advice to improve subsequent calls.

• Reward Guided Refine Loop: Trace how Refine calls a supplied submodule up to N times, scores each response with a reward function, and returns early when a response exceeds the threshold.
  Why it matters: The reward and threshold determine whether Refine accepts an answer or spends another attempt improving it.
• Grounded Feedback and Hints: Explain how an unsuccessful response leads to an LM call for advice, and how that advice becomes a hint on the next call to the supplied program.
  Why it matters: The feedback call explains how a low-scoring attempt can change the next attempt rather than merely repeat it.
• Per Query Hint Optimization: Distinguish Refine’s per-query hints from the instruction updates made by the chapter 6 optimizers.
  Why it matters: This distinction clarifies what Refine adapts for an individual query versus what remains fixed after optimization.
• Source Based Prompt Grounding: Describe how source code and module summaries can be collected as text to help an LM propose prompt improvements.
  Why it matters: Giving the LM relevant code and module descriptions can help it suggest better prompt changes.

Key terms: Refine, submodule, reward function, N, threshold, Predict, trajectory, discussion, advice, hint

7.4 Examples of custom modules

This section introduces custom DSPy modules as a way to implement application-specific workflows or established prompting techniques and test whether they improve performance.

• Purposes of custom modules: Explain why a DSPy application might need a custom module: to code a workflow specific to the application or implement a prompting technique, then determine whether it improves performance.
  Why it matters: A custom module is useful when it lets you try an approach that fits your application and check whether it helps.

Key terms: custom modules, workflows specific to your application, standard prompting techniques, performance

7.4.1 Adding a closing section

This section shows how a custom module can add text to prompt instructions and explains why such additions should remain a baseline rather than become extensive manual prompt tuning.

• Adding text to instructions: Explain that this prompting technique adds content to a prompt’s instructions rather than changing its fields, and that the added content can be a prompt component or other instructional text.
  Why it matters: It identifies what the custom module changes in the prompt.
• Manual wording versus optimization: Justify using added instruction text as a possible stronger baseline without spending too much effort manually tuning individual phrases, since an optimizer may rewrite the instructions or identify useful phrases.
  Why it matters: It keeps a useful starting prompt from turning into an open-ended manual tuning exercise.

Key terms: instructions, fields, prompting techniques, baseline, optimizer, prompt engineering

7.4.2 Rewording the inputs

Shows how a custom DSPy module can reword an input before answering it, using separate LMs while managing the cost of the extra step.

• Two-step input rewording: Trace how the module first makes a question easier to answer and then passes the reworded question to an answering LM. Explain why separating rewording from answering can make each task simpler.
  Why it matters: Rewording a difficult input first can leave the answering LM with a clearer, more manageable task.
• Different LMs for each step: Explain why the example uses gpt-4o to shorten the question and gpt-4o-mini to answer it.
  Why it matters: The answering step need not use the same LM as the input-processing step.
• Extra-step cost management: Describe the potential cost of adding an input-processing step and the ways the section suggests moderating it.
  Why it matters: The benefit of a clearer input has to be weighed against the additional LM call.

Key terms: two-step method, submodules, cleaning, gpt-4o, gpt-4o-mini, two steps, lower-cost LMs, inputs and outputs

7.4.3 Separating out toxic comments

This subsection shows how a custom module can check user input for toxicity before classification and explains why separating that check is useful.

• Toxicity check before classification: Explain why a custom module checks whether a user utterance is toxic before attempting to classify it, and describe how toxic content might be handled differently.
  Why it matters: Checking first prevents potentially invalid user input from being treated like an ordinary classification request.
• Separate toxicity model choice: Explain how separating the toxicity check from classification allows a model especially strong at toxicity detection to be used, while noting the model choice in this example.
  Why it matters: Separating the tasks leaves room to choose a better-suited model for the toxicity check.

Key terms: toxic, custom module, utterance, classify, LM, toxicity checks, submodules, gpt-4o-mini

7.4.4 Module for checking security

The section shows how a custom DSPy module can screen user messages with reusable security checks before running its main task.

• Prequery security screening: Explain how dedicated DSPy programs can check a user message for prompt injection or PII before the main submodule runs, and why this simple approach is framed for lower-risk situations.
  Why it matters: Screening before the main task gives the application a place to check potentially unsafe input.
• Independent reusable check programs: Justify optimizing threat detection separately from question answering, and describe how a shared output field lets a custom module accept different screening programs.
  Why it matters: Separate, compatible checks can be optimized for their own purposes and reused across applications.
• Sensitive data screening location: Explain why a self-hosted LM could be used to check for sensitive information before data is passed to a remote LM, while distinguishing that option from what the example does.
  Why it matters: Where a sensitive-data check runs matters if the input might otherwise reach a remote LM.

Key terms: prompt injections, personally identifiable information (PII), submodule, main submodule, security_prog, pii_prog, has_problem, preoptimized DSPy programs, PII, self-hosted LM

7.4.5 Chain of Draft

This section explains how Chain of Draft keeps the potential benefit of a reasoning rationale while reducing its length and output-token overhead through a custom ChainOfThought rationale field.

• Concise rationale tradeoff: Explain why Chain of Draft requests a brief rationale rather than the potentially long rationale produced by Chain of Thought, and how this can reduce output tokens while retaining a useful reasoning step.
  Why it matters: Shorter reasoning can make repeated LM requests less expensive without simply removing the rationale.
• Custom rationale field limit: Describe how the example implements Chain of Draft by using ChainOfThought as a submodule and replacing its rationale field with one that instructs it to use at most five words. Explain that rationale instructions could instead request more detail, increasing tokens but possibly improving accuracy or creativity.
  Why it matters: The rationale field gives you a direct way to choose between concise reasoning and potentially more useful detail.

Key terms: Chain of Draft, Chain of Thought, rationale, output tokens, ChainOfThought, rationale_field, five words

7.4.6 Creating a critic-based module

This section explains how a critic-based custom module improves a response through a loop of execution, critique, and prompt revision, and weighs that approach against optimizing a prompt once.

• Critic based revision loop: Trace how CriticModule executes a query, critiques its response, and proposes a better prompt, repeating until the critic approves the response or the maximum number of tries is reached.
  Why it matters: The loop shows how a custom module can use feedback to improve its response during a single query.
• Per query prompt optimization tradeoff: Distinguish CriticModule's prompt search during each query from optimizing a simple module's prompt once, and explain when the added production cost may be worthwhile.
  Why it matters: This tradeoff helps determine whether query-specific improvement justifies repeated LM calls in production.
• Nested custom modules: Describe how this example composes custom modules, including its use of ChainOfDraft as a submodule.
  Why it matters: Nesting lets a custom module reuse another module as part of a larger workflow.

Key terms: CriticModule, Critic, prompt_writer, critique object, inference time, optimizer, Predict, production, custom module, submodule

7.5 Parallelism in custom modules

The section introduces two kinds of DSPy parallelism and explains when their potential speed gains justify the complexity of asyncio.

• Two forms of parallelism: Distinguish running multiple DSPy programs at the same time from calling multiple submodules or tools at once within a module, and state that both use Python’s asyncio library.
  Why it matters: These are the two places you can look for parallel execution in a DSPy application.
• Asyncio adoption tradeoff: Justify starting with synchronous code and switching to asyncio only when identified slow parts warrant its added complexity.
  Why it matters: The potential speedup is most useful when it outweighs the extra development and debugging costs.

Key terms: parallelism, DSPy programs, submodules, tools, asyncio, synchronous, asynchronous processing, error handling, debugging

7.5.1 The asyncio library

This section explains how asyncio uses coroutines and an event loop to overlap waiting tasks, then shows how to start and await concurrent tasks.

• Single-threaded concurrency: Explain how asyncio can overlap waiting tasks without executing two lines of Python code at the same time, and why this suits LM calls better than CPU-heavy processing.
  Why it matters: This explains when concurrency can make an LM application faster.
• Coroutines and event-loop switching: Describe how async def, await, tasks, and the event loop work together to pause and resume coroutines.
  Why it matters: These are the pieces that let waiting operations share execution time.
• Event-loop startup and notebooks: Explain how asyncio.run() starts an async coroutine and why running asyncio code in a notebook may require nest_asyncio.apply().
  Why it matters: It helps explain why the same async example can be simpler to run in a .py file than in a notebook.
• Concurrent task creation and results: Describe how gather() or create_task() creates tasks for concurrent execution, and how awaiting them affects completion and returned values.
  Why it matters: It shows how to launch independent operations together without losing track of their results.
• Awaitable waits and elapsed time: Explain why the example uses asyncio's awaitable sleep and why its concurrent five- and three-second waits take about five seconds overall.
  Why it matters: The timing example makes the benefit of overlapping waits observable.

Key terms: asyncio, asynchronous, concurrent, thread, coroutine, async def, await, task, event loop, asyncio.run()

7.5.2 Calling modules in parallel

This section explains how to call DSPy modules concurrently, what custom modules need to support asynchronous calls, and the benefits and limits of doing so.

• Parallel calls and tradeoffs: Explain why executing multiple LM tasks in parallel can save time and why the number of simultaneous calls may need to be limited.
  Why it matters: Parallel calls can improve response time, provided the service can handle the simultaneous requests.
• Asynchronous custom module calls: Describe how a module call becomes an asyncio task and what a custom module must provide to support parallel execution.
  Why it matters: These method roles let a custom module support both ordinary calls and concurrent calls.

Key terms: parallel, concurrent LM calls, rate limits, acall(), aforward(), forward(), async, await, asyncio task

7.5.3 Parallel execution within modules

This section explains how to run awaitable work concurrently inside a DSPy module, using candidate-answer generation as an example.

• Concurrent work inside modules: Explain how a DSPy module can create and await concurrent tasks, and describe how the example uses those tasks to gather candidate answers before selecting the best one.
  Why it matters: Concurrency can reduce the time spent collecting independent candidate answers within a module.

Key terms: concurrent execution, gather(), create_task(), asyncio tasks, await, submodules

7.5.4 Asynchronous Skeleton-of-Thought

The section shows how an asynchronous Skeleton-of-Thought module splits a question into independent jobs, runs them concurrently, and combines their answers.

• Skeleton-of-Thought task decomposition: Explain why Skeleton-of-Thought divides a problem into subtasks and how far the example takes that division.
  Why it matters: The decomposition explains both why the LM receives smaller jobs and why those jobs can be run concurrently.
• Three-module execution flow: Trace how the three modules turn a query into a combined result.
  Why it matters: The sequence distinguishes parallel work on the smaller jobs from the final step that assembles their answers.
• Asynchronous implementation choices: Explain why the example uses create_task() and what role nest_asyncio plays.
  Why it matters: These choices explain how the example accommodates a variable number of jobs and certain execution environments.

Key terms: Skeleton-of-Thought, subtasks, independent, parallel, submodule, concurrently, awaits, combine, create_task(), gather()

7.6 Optimizing custom modules

This section establishes that custom DSPy modules can be optimized like built-in modules and identifies the setup needed for the examples.

• Custom module optimization: Explain that custom DSPy modules can be optimized in the same way as built-in modules.
  Why it matters: You can apply DSPy optimization to modules you create yourself, not just its built-in modules.

Key terms: custom modules, built-in modules, optimized

7.6.1 Using BootstrapFewShot

Shows how BootstrapFewShot creates demonstrations for each step of a custom module from successful runs of the whole program.

• Whole-program bootstrapping of submodules: Explain how BootstrapFewShot uses a successful run of a multistep custom module to create demonstrations for its submodules, including generated intermediate values.
  Why it matters: A metric on the final result can supply demonstrations for every step of a custom module.
• Two-step optimization example: Describe the two-submodule example and how its compiled prompts are inspected.
  Why it matters: The example makes it possible to see that one selected run supplies demos to both prompts.

Key terms: BootstrapFewShot, bootstrapped demos, metric function, submodules, clear_question, custom module, prog, inspect_history()

7.6.2 Optimizing using MIPROv2

This section explains how MIPROv2 optimizes a custom module’s submodules and why joint optimization can be preferable to optimizing prompts one at a time.

• MIPROv2 optimization stages: Describe how MIPROv2 creates bootstrapped demos and candidate instructions for each submodule, then selects a combination of demos and instructions.
  Why it matters: It clarifies what MIPROv2 changes when optimizing a module with multiple predictors.
• Joint submodule optimization: Distinguish MIPROv2’s optimization of prompts together from COPRO’s one-prompt-at-a-time approach, and explain why optimizing together can help when submodule calls depend on one another.
  Why it matters: It explains why optimizer choice matters for a custom module whose submodules work in sequence.

Key terms: MIPROv2, bootstrapped demos, instructions, submodule, COPRO, submodules, LM calls

7.6.3 Examining the set of submodules

This subsection shows how to inspect the Predict modules inside a nested DSPy module, including after optimization.

• Nested predictor inspection: Explain how named_predictors() helps inspect a nested module by listing its Predict modules and their details, and how running it again after optimization helps reveal what changed.
  Why it matters: It gives you a way to see which predictors a complex module contains and what optimization changed.

Key terms: named_predictors(), Predict modules, signature, instructions, fields, compilation

7.7 Creating complex vs. simple modules

This section explains when to put a multi-call workflow in one custom DSPy module and when to keep its programs separate.

• Whole-workflow custom module: Explain why a repeated multi-call workflow is generally better encapsulated in one custom module than coordinated through separate Python code.
  Why it matters: This is the section’s default design choice for a workflow that needs multiple LM calls.
• Reasons to separate programs: Describe the exceptions to grouping a workflow in one module: useful intermediate results or programs that are simpler or more efficient to optimize separately.
  Why it matters: These exceptions keep the default recommendation from becoming a rigid rule.

Key terms: custom module, submodules, full workflow, intermediate results, optimize the programs separately

7.8 Using DSPy to create custom modules

This section explains how DSPy can generate custom-module code from a workflow description, how to test and improve that code, and when the approach is worth using.

• Custom Module Code Generation: Describe the two ways to use DSPy to create custom-module code: ask it to improve a module you wrote, or give it a general workflow description and ask it to generate the module.
  Why it matters: You can start with a description of the workflow instead of implementing every submodule yourself.
• Generated Code Validation: Explain how the generated code is checked and distinguish the example's execution-only test from a meaningful performance evaluation.
  Why it matters: Code that runs without errors may still be ineffective at the task.
• Iterative Code Improvement: Trace how repeated generation and testing can use coding errors and validation results to improve the next set of instructions, and distinguish that process from later prompt optimization.
  Why it matters: Feedback can improve unreliable generated code before prompt optimization is applied to the resulting module.
• Generated Workflow Descriptions: Describe how DSPy can first generate a workflow description and then use it as the code description for generating a custom module.
  Why it matters: DSPy can help design the workflow as well as write its implementation.
• Appropriate Use of Generation: Justify when generating DSPy modules through this process may be worthwhile despite its cost and variable results.
  Why it matters: Good examples and a metric make an expensive generation-and-testing effort more defensible.

Key terms: custom module, workflow, powerful LM, validation set, metric function, exec, python_code_content, coding errors, instructions, optimizers

7.9 Summary