liveBook · Manning

Reward signal

As explained in Deep Learning with Python, Third Edition

Feedback value used in reinforcement learning to indicate how desirable an action or output is.

A reward signal turns a model’s output into feedback: higher values mark responses judged more desirable, while lower values indicate what to avoid. section 16.4.1

Filtering generated responses

A model generates responses, a separate reward model assigns each one a floating-point score, and responses above a cutoff become training data. Practical RLHF usually uses all scores to steer updates, because poor responses also signal what not to do. section 16.4.1

Listing 16.15 Pseudocode for the simplest possible RLHF algorithm
for prompts in dataset:
    responses = model.generate(prompts)                               #1
    rewards = reward_model.predict(responses)                         #2
    good_responses = []
    for response, score in zip(responses, rewards):
        if score > cutoff:
            good_responses.append(response)
    model.fit(good_responses)                                         #3

Trace the minimal loop from generation through reward prediction and cutoff-based selection. section 16.4.1

How Deep Learning with Python, Third Edition covers it

How deep it goes
Defines scalar response scoring and sketches filtering, while deferring specialized gradient-update algorithms and deeper reinforcement-learning details.
Examples
Demonstrates prompt-response scoring, cutoff filtering, and iterative regeneration with improved reward models.
What it assumes
Assumes familiarity with prompts, generated responses, and model training; introduces actions, rewards, and reward-model scoring.
What you can do after
Apply it via a reward model to score generated responses and select or steer training examples.
How it connects
Connects reward signals with reinforcement learning, reward models, human preference rankings, supervised fine-tuning, and language-model generation.
Cover of Deep Learning with Python, Third Edition
liveBookmanning.com
Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 11 books explain reward signal →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.