liveBook · Manning

Function approximator

As explained in Applied Reinforcement Learning

Function approximation estimates a value function or policy with a parameterized mathematical function instead of storing a separate value for every state or state-action pair.

A function approximator estimates values from a description of a state, rather than looking up a separate stored value for every state or state–action pair. Its adjustable parameters act like knobs: changing them can affect estimates for many states at once, allowing the agent to generalize. section 8.1

Radiotherapy dose planning

A treatment state can include tumor volume, biological indicators, accumulated dose, and patient-specific factors. Because these variables produce too many possible states to enumerate, an approximator can estimate the value of dose adjustments from the state description instead of relying on a complete table. section 8.0 section 8.1

In value-based reinforcement learning, Q̂(s,a;θ) denotes an estimated value for taking action a in state s, with parameters θ. Learning adjusts those parameters rather than filling individual table entries. Temporal-difference error can drive those adjustments. section 8.6 section 8.2

The choice of approximator depends on the available state representation:

  • A linear model fits when useful, interpretable features can be designed explicitly. section 8.6 section 8.2
  • A deep neural network can learn representations from raw inputs such as images or sensor vectors. Its layers combine weighted transformations with nonlinear activations to model more complex relationships. section 8.2 section 8.6
Figure 8.4 End-to-end learning in deep neural networks

Follow the deep model’s learning loop from input and prediction to parameter updates. section 8.2

A deep network can approximate Q-values in a Deep Q-Network, but the added flexibility brings training challenges. Changing bootstrap targets and correlated sequential data can destabilize learning. Experience replay samples stored transitions to break correlations, while a target network holds parameters fixed for a period to stabilize targets. section 8.6

How Applied Reinforcement Learning covers it

How deep it goes
Explains parameter adjustment with temporal-difference error and traces deep-network learning through layers, nonlinear activations, and backpropagation.
Examples
Uses radiotherapy dose planning with tumor volume, biological indicators, accumulated dose, and patient-specific factors as state variables.
What you can do after
Evaluate/understand when linear models with engineered features or deep networks for raw inputs are appropriate.
How it connects
Connects value functions with temporal-difference learning, deep Q-networks, experience replay, and target networks.

Also explained in

Compare how 3 books explain function approximator →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.