liveBook · Manning

Value-based method

As explained in Applied Reinforcement Learning

A value-based method learns action or state values and derives the policy by selecting actions with the highest estimated value.

A value-based method estimates how valuable each action is in a state, then derives a policy by choosing the action with the highest estimated value. This works naturally with discrete actions, but continuous controls must be discretized. section 9.1

The data center control problem

Eight zones each have a fan speed and valve opening between 0 and 1. Discretizing each control into just five levels creates 5^16, or about 152 billion, possible actions—too many for a Q-network output. Coarse choices also miss nuanced settings, such as a fan speed of 0.73. section 9.1

Value-based methods also produce deterministic choices: for a given state, they select the highest-valued action. ε-greedy exploration adds noise, but is described here as a heuristic rather than a principled part of the method. section 9.1

Figure 9.1 Value-Based vs. Policy-Based Reinforcement Learning.

Compare value-based and policy-based reinforcement learning. section 9.1

How Applied Reinforcement Learning covers it

How deep it goes
Defines value-based action selection and works through discretization limits, without detailing algorithm internals.
Examples
Uses data center controls, gas station inventory, radiotherapy planning, and vehicle routing as examples.
What you can do after
Evaluate when discrete-action value estimates become impractical for continuous or high-dimensional controls.
How it connects
Connects value-based methods to Q-learning, DQN, bandits, and policy-gradient methods.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.