A function approximator estimates values from a description of a state, rather than looking up a separate stored value for every state or state–action pair. Its adjustable parameters act like knobs: changing them can affect estimates for many states at once, allowing the agent to generalize. section 8.1
Radiotherapy dose planning
A treatment state can include tumor volume, biological indicators, accumulated dose, and patient-specific factors. Because these variables produce too many possible states to enumerate, an approximator can estimate the value of dose adjustments from the state description instead of relying on a complete table. section 8.0 section 8.1
In value-based reinforcement learning, Q̂(s,a;θ) denotes an estimated value for taking action a in state s, with parameters θ. Learning adjusts those parameters rather than filling individual table entries. Temporal-difference error can drive those adjustments. section 8.6 section 8.2
The choice of approximator depends on the available state representation:
- A linear model fits when useful, interpretable features can be designed explicitly. section 8.6 section 8.2
- A deep neural network can learn representations from raw inputs such as images or sensor vectors. Its layers combine weighted transformations with nonlinear activations to model more complex relationships. section 8.2 section 8.6
Figure 8.4 End-to-end learning in deep neural networks
Follow the deep model’s learning loop from input and prediction to parameter updates. section 8.2
A deep network can approximate Q-values in a Deep Q-Network, but the added flexibility brings training challenges. Changing bootstrap targets and correlated sequential data can destabilize learning. Experience replay samples stored transitions to break correlations, while a target network holds parameters fixed for a period to stabilize targets. section 8.6


