liveBook · Manning

Model-based approach

As explained in Applied Reinforcement Learning

A model-based method solves a decision problem by using a known or learned model of the environment to plan ahead.

A model-based approach plans ahead using a representation of how an environment works. In reinforcement learning, that model combines transition probabilities with a reward function; it may be known or learned. section 4.0

For business problems, a model can be built from carefully chosen assumptions, parameters, and expert knowledge. Data-driven methods instead learn patterns from historical data, and combining the two approaches can be practical. section 1.2

The transition function determines how the method plans:

  • When the transition function is known, dynamic programming offers a direct way to solve the decision problem. section 4.0
  • Methods that bypass the transition function learn from sampled state, action, reward, and next-state experiences; this can require more interaction, computation, and time. section 4.0

How Applied Reinforcement Learning covers it

How deep it goes
Defines the environment model and contrasts known, learned, and omitted transition functions, without detailing model-learning procedures.
What it assumes
Builds on representing sequential decisions as Markov decision processes and distinguishing known from learned transition functions.
What you can do after
Evaluate when a known or learned environment model supports planning and recognize the costs of bypassing the transition function.
How it connects
Connects model-based methods to Markov decision processes, dynamic programming, and model-free methods.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.