liveBook · Manning

Reward signal

As explained in Applied Reinforcement Learning

A reward is a numerical feedback signal indicating the desirability of an outcome or transition after an agent's action.

A pricing agent needs feedback after it sets a price. Daily profit can provide that feedback as a reward signal: a single number indicating how desirable an outcome was. Rewards steer the agent toward decisions that improve cumulative outcomes, not just immediate gains. section 2.4 section 1.1

In a Markov decision process, an agent chooses an action in its current state, receives a reward, and moves to a next state. The action affects both the immediate reward and the probabilities of future states. Learning which earlier decisions contributed to success or failure is the credit-assignment challenge. section 2.3 section 1.1

Reward engineering translates business goals into a signal an agent can learn from. A useful design may combine objectives such as profit and service quality with penalties for undesirable outcomes. Give feedback at appropriate steps and watch for mismatched scales between terms. For hard limits, use action masking alongside penalties; soft constraints can be handled with penalties and trade-offs. section 2.6 section 2.7

Rewards for business decisions

An online store can reward daily profit: sales revenue minus associated costs. A call center can instead penalize customer waiting time and labor expense. In an ad campaign, the signal can be simpler: 1 for a click and 0 for no click; the agent uses observed clicks to learn which ads perform well. section 2.4 section 5.3

Table 2.1 Comparing Different Business Problems Modeled as Markov Decision Processes
Example
State design
Action design
Reward design
Non-Makrov if…
Dynamic Pricing in Online Store
Inventory level, current price, demand forecast, days left in month
Set daily price
Daily profit (revenue – costs)
Ignoring inventory level or pending backorders
Workforce Scheduling in Call Center
Queue length, available agents, forecasted call arrivals
Choose number of agents to schedule
Negative cost: waiting time penalties + salaries
Ignoring agent fatigue or cumulative workload history
Preventive Maintenance for Machines
Machine condition, time since last maintenance, environmental factors
Choose whether to maintain each machine
Cost of maintenance vs cost of breakdowns
Omitting time since last maintenance or operational history
Customer Retention Campaign
Active status, weeks since last engagement, recent activity, subscription tenure
Choose between offer, reminder, or silence
+1 for retention, 0 for churn
Missing recent activity history or engagement recency

Compare how rewards, states, and actions differ across business decisions. section 2.4

A reward can also encourage the wrong behavior. Paying for machine runtime may produce unwanted inventory, while rewarding clicks alone can favor clickbait over sales and user satisfaction. Choose proxies carefully, combine measures when appropriate, and use simulations, audits, and scenario testing to look for shortcuts. section 2.6.6

The algorithm does not decide the business objective for you. For example, a portfolio reward must balance returns and risk without inadvertently encouraging the agent never to invest. Reward design is part of designing and evaluating the whole decision system. section 12.2

How Applied Reinforcement Learning covers it

How deep it goes
Explains reward engineering through timing, objective and constraint terms, scale, and checks for unintended shortcuts.
Examples
Illustrates rewards with daily pricing profit, call-center costs, binary ad clicks, and recommendation clickbait.
What you can do after
Evaluate reward designs for alignment with business objectives, constraints, and unintended behavior.
How it connects
Connects reward signals to Markov decision processes, policies, credit assignment, action masking, and multi-armed bandits.

Also explained in

Compare how 16 books explain reward signal →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.