A value-based method estimates how valuable each action is in a state, then derives a policy by choosing the action with the highest estimated value. This works naturally with discrete actions, but continuous controls must be discretized. section 9.1
The data center control problem
Eight zones each have a fan speed and valve opening between 0 and 1. Discretizing each control into just five levels creates 5^16, or about 152 billion, possible actions—too many for a Q-network output. Coarse choices also miss nuanced settings, such as a fan speed of 0.73. section 9.1
Value-based methods also produce deterministic choices: for a given state, they select the highest-valued action. ε-greedy exploration adds noise, but is described here as a heuristic rather than a principled part of the method. section 9.1
Figure 9.1 Value-Based vs. Policy-Based Reinforcement Learning.
Compare value-based and policy-based reinforcement learning. section 9.1
