MIPROv2

As explained in Building LLM Applications with DSPy

MIPROv2 is a DSPy optimizer that uses dataset summaries and grounding information to guide prompt proposals and includes demonstrations in task prompts.

MIPROv2 is a DSPy optimizer for finding prompt instructions and demonstrations together. Tuning them jointly matters because an effective instruction can depend on its examples, and effective examples can depend on the instruction. It offers an alternative to manually revising a prompt one part at a time. section 6.2

Optimization has three stages:

• Build candidate demonstration sets by repeatedly bootstrapping from training examples. Include an empty set so zero-shot prompting remains an option. section 6.2
• Ask a prompt LM to propose candidate instructions. Grounding gives it more task information to guide those proposals. section 6.2
• Try combinations of instructions and demonstrations against validation examples. Bayesian optimization uses earlier scores to choose promising combinations rather than testing every pair. section 6.2 section 6.2.2

Figure 6.2 The MIPROv2 algorithm step 3. In steps 1 and 2, we create the candidate sets of demos and candidate instructions. In step 3, we try various ways to combine them until we find the combination that performs best. Because there are often many combinations, this step uses a Bayesian optimization method to determine which combinations to try. In this example, we try set of demos #5 and instructions #3, creating a prompt that is tested on all or part of the validation set with the task LM. The score is recorded, allowing us to track which sets of demos work well, which instructions work well, and which work well together. This helps us determine which other combinations to test in subsequent iterations.

The diagram shows how a selected instruction and demonstration set become a prompt, receive a validation score, and inform later trials. section 6.2

Optimize a flight-intent classifier

Create `dspy.MIPROv2(metric=validate_answer, auto="light")`, then compile the classifier with `trainset`, `valset`, `max_bootstrapped_demos=1`, and `max_labeled_demos=4`. One recorded trial scored 34 of 35 examples on a validation minibatch; that trial score is not a guarantee of performance on new inquiries. A separate prompt model can be supplied with `prompt_model`. section 6.2

The search also applies to programs with multiple submodules: MIPROv2 proposes instructions for each and optimizes their prompts together. This differs from COPRO’s one-prompt-at-a-time approach and is especially relevant when one LM call depends on another. section 7.6.2

Search still has a cost. In the flight-intent example, a run took about two minutes, 424,000 tokens, and 675 LM calls on Google Colab. Bayesian trial selection helps avoid exhaustive testing, but the best optimizer can still depend on the task and LM. section 6.2 section 6.9

How Building LLM Applications with DSPy covers it

How deep it goes

Explains three search stages and Bayesian trial selection, without deriving the optimization mathematics.

Examples

Uses a flight-intent classifier, a scored validation minibatch, and a multihop module.

What it assumes

Assumes a DSPy program, task metric, and training and validation sets; introduces the search process without requiring Bayesian optimization mathematics.

What you can do after

Apply it via DSPy's `dspy.MIPROv2` to compile a program against training and validation sets with a task metric.

How it connects

Connects BootstrapFewShot, COPRO, grounding, few-shot prompting, and Bayesian optimization.

liveBookmanning.com

Read Building LLM Applications with DSPy on liveBook Get the book at manning.com

Related concepts

• BootstrapFewShotuses · BootstrapFewShot is a DSPy optimizer that selects or bootstraps demonstrations for a prompt from training examples and successful model-generated outputs.
• groundingused by · Grounding is the practice of anchoring or constraining an AI system’s outputs with relevant information, context, or evidence.
• Optunauses · Optuna is an open-source Python framework for automated hyperparameter optimization through repeated trials that explore parameter configurations.
• COPROcontrasts with · COPRO is a DSPy prompt-optimization technique that iteratively generates and evaluates alternative prompt instructions to improve their performance.
• Bayesian optimizationuses · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
• optimizerkind of · An optimizer is an algorithm or procedure that adjusts model parameters or other decisions to improve an objective function.

This explainer was generated by Manning's knowledge graph from the text of Building LLM Applications with DSPy. The section links point to the parts of the book it draws on.