liveBook · Manning

Reward function

As explained in Applied Reinforcement Learning

A reward function is a mapping that assigns numerical feedback to states, actions, transitions, or outcomes in a decision process.

A reward function assigns numerical feedback to decisions and their outcomes. In reinforcement learning, it translates goals into a signal that steers an agent’s behavior. For a factory, that signal might combine sales revenue with the costs of production, inventory, and unmet orders. section 2.5 section 3.1

A reward can depend on the current state, action, and resulting state. An environment typically calculates it as part of a step, alongside the transition to the next state. Rewards accumulated over a trajectory let policies be compared over time, rather than judging a single decision in isolation. section 3.1 section 2.5 section 4.4

Scoring a production day

A factory chooses whether to produce scooters, produce bikes, or idle. Its daily reward adds revenue from fulfilled demand and subtracts production, holding, setup, and backorder costs. Switching products incurs a $200 setup cost, so a change must be worthwhile enough to offset it. The score captures trade-offs among sales, inventory, and service rather than rewarding production volume alone. section 2.5

Listing 2.7 Reward function
def reward_function(state, action, demand,params:FactoryProblemDefinition,
    produced_scooter, produced_bike, \\
                backorder_scooter, backorder_bike, next_state):
 
    _day, _inv_scooter, _inv_bike, current_setup = state        #A
    _next_day, next_inv_scooter, next_inv_bike, _next_setup = next_state #A
    demand_scooter, demand_bike = demand                                 #A
 
    revenue = (demand_scooter - backorder_scooter) * \\
             params.price_scooter + (demand_bike - backorder_bike) * \\
                        params.price_bike         #B
 
    production_cost = produced_scooter * params.cost_scooter +       #C
              produced_bike * params.cost_bike                       #C
 
    setup_penalty = params.setup_cost \\                               #D
              if (action in (0, 1) and action != current_setup) else 0 #D
 
    holding_penalty = params.holding_cost * \\
                           (next_inv_scooter + next_inv_bike)  #E
 
    backorder_penalty = params.backorder_cost * \\                #F
              (backorder_scooter + backorder_bike)                 #F
 
    total_reward = revenue - production_cost - \\                 #G
              setup_penalty - holding_penalty - backorder_penalty #G
 
    return total_reward

Inspect how the factory reward uses demand, production, setup, backorders, and the next state. section 2.5

Practical reward design starts with the business decision:

  • Identify the objectives and translate them into measurable reward terms. Include costs and undesirable outcomes, not just the activity the agent should increase. section 2.6 section 2.5
  • Choose when feedback arrives and balance term scales. Stepwise feedback can guide intermediate behavior, while a terminal reward or penalty can reflect the final outcome. section 2.7 section 3.3
  • Distinguish hard constraints from soft trade-offs. Use action masking alongside penalties for hard constraints; soft constraints can be handled through penalties and dynamic trade-offs. section 2.7
  • Test what the agent actually learns. Simulations, audits, and scenario tests can reveal shortcuts that improve the score without advancing the real objective. section 2.6.6

Useful output versus a proxy

Rewarding machine runtime alone can encourage a machine to run continuously and create unwanted inventory. Similarly, paying for recommendation clicks can favor clickbait over sales and user satisfaction. Combining clicks with revenue and a penalty for bounces is one way to make the signal reflect more of the intended outcome. section 2.6.6

Warehouse and pricing rewards

A warehouse robot receives a small penalty for each valid move, larger penalties for walls and obstacles, rewards for pickups, and a large reward for returning to the depot after collecting everything. In perishable pricing, each period’s sales revenue is rewarded, while leftover inventory receives a terminal penalty. Both designs balance immediate feedback with completion of the wider task. section 3.2 section 3.3

Table 3.1 Key reward signals used in warehouse order picking environment
Agent behavior
Reward
Purpose
Move to a valid (empty) cell
–0.1
Small step penalty to encourage shorter, more efficient paths
Hit wall (outside grid)
–0.5
Penalize invalid moves outside the grid limits
Attempt to move into a blocked cell (hit obstacle)
–0.7
Stronger penalty to discourage unsafe or infeasible paths
Successfully pick up an item
+15.0
Significant reward for achieving one of the primary objectives
Return to the depot after collecting all items
+100.0
Large terminal reward for completing the full task successfully
Episode ends without full completion (e.g., time limit reached)
–20.0
Penalty for failing to complete the task within the allowed steps

Compare the warehouse reward values and the behavior each is intended to encourage. section 3.2

Not every useful reward is an explicit business formula. For language-model feedback, qualities such as helpfulness and safety can be difficult to express directly, so a reward can be learned from human preferences and applied to a completed response. For verifiable answers, a verifier can instead extract an answer and compare it with a reference to produce a scalar score. section 10.3 section 11.7.4

How Applied Reinforcement Learning covers it

How deep it goes
Develops reward components, timing, constraints, scale, and shortcut checks through implemented environments and worked examples.
Examples
Uses factory production, warehouse picking, perishable pricing, resource allocation, recommendations, and language-model answer verification.
What you can do after
Build reward functions from scratch for modeled decisions and test whether their signals encourage intended behavior.
How it connects
Connects reward functions to Markov decision processes, policies, transition dynamics, action masking, and reinforcement learning from human feedback.

Also explained in

Compare how 3 books explain reward function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.