Reward function

As explained in Building LLM Applications with DSPy

A reward function is a callable that evaluates a response and returns a score, with higher scores indicating better responses.

A reward function scores a generated response so a DSPy module can decide whether to keep it or try again. BestOfN compares responses from repeated calls to one submodule; Refine uses scores to guide successive attempts. Higher scores indicate better responses, and a threshold can end the search early. section 7.3.2 section 7.3.4

The callable receives the inputs and a prediction, then returns a numeric score. It runs at inference time, so it cannot rely on a known correct output as a metric sometimes can. Choose a rule that can judge the response using information available when the program runs. section 7.3.2

Reward a one-word answer

For a question-answering program, `def one_word_answer(args, pred): return 1.0 if len(pred.answer.split()) == 1 else 0.0` scores the desired format without needing a ground-truth answer. Pass it as `reward_fn` to `dspy.Refine` with `N=3` and `threshold=1.0`. Refine can stop as soon as an answer earns 1.0; otherwise, it uses feedback to improve its next attempt. section 7.3.4

Figure 7.1 The workflow within the Refine module starts with the original prompt, which is used in a call to the LM. The response is collected, and Refine calls the reward method to evaluate it. At each iteration, the LM is also called to get feedback about the prompt and the response, including advice on how to improve the prompt. When asking for this advice, Refine also provides the LM with task and reward function information, a history of the previous prompts, and the reward function’s scores for their outputs. This advice is used as the hint field in the next call to the LM. Once the response is good enough (the reward function gives it a score above a specified threshold) or the process loops a specified number of times, the best response (as evaluated by the reward function) is returned.

The workflow shows where scoring fits: Refine evaluates each response, uses unsuccessful attempts to seek feedback, and returns the best response when it reaches the threshold or iteration limit. section 7.3.4

How Building LLM Applications with DSPy covers it

How deep it goes

Shows a callable signature and simple scoring rules, then traces how modules use scores; does not develop a general scoring-design procedure.

Examples

Uses a negative answer-length score and a one-word-answer score for a question about Belgium.

What it assumes

Assumes basic Python functions and DSPy modules; demonstrates the scoring callable with short examples.

What you can do after

Apply it via DSPy BestOfN or Refine to score responses and stop when a satisfactory score is reached.

How it connects

Connects reward functions to BestOfN, Refine, prediction modules, metrics, thresholds, and inference-time evaluation.

liveBookmanning.com

Read Building LLM Applications with DSPy on liveBook Get the book at manning.com

Related concepts

• Group Relative Policy Optimizationused by · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
• Refineused by · Refine is a DSPy module that iteratively evaluates responses and uses feedback to improve subsequent outputs.
• reinforcement learning with verifiable rewardsused by · Reinforcement learning with verifiable rewards is a training approach that optimizes a model using rewards derived from objectively checkable outcomes.
• JSONuses · JSON is a standardized, language-independent text format for representing structured data using objects, arrays, strings, numbers, Boolean values, and null.

This explainer was generated by Manning's knowledge graph from the text of Building LLM Applications with DSPy. The section links point to the parts of the book it draws on.