liveBook · Manning

Model-free method

As explained in Applied Reinforcement Learning

Model-free learning is reinforcement learning in which an agent learns action selection directly from experience without an explicit environment transition model.

A model-free method learns to choose actions from experience without an explicit model of how the environment changes. Each experience provides a state, action, reward, and next state; the agent uses these sampled transitions rather than known transition probabilities. section 4.0 section 6.7

This approach fits problems where the environment’s rules are hidden or difficult to obtain. It avoids requiring a complete transition model, but relies on more interactions, computation, and time; rewards remain essential. section 4.0 section 4.5

Without transition dynamics, state values alone do not identify which action to take. Model-free agents can learn action values, Q(s,a), directly from sampled experience. Each sample is noisy evidence, and tracking values for state–action pairs makes learning more complex. section 6.3 section 6.3

Temporal-difference learning, Q-learning, and SARSA are model-free methods. SARSA is on-policy, while Q-learning is off-policy. section 6.7

How Applied Reinforcement Learning covers it

How deep it goes
Explains experience tuples, direct Q-value learning, and sampling trade-offs, while leaving specific update rules to related methods.
Examples
Uses uncertain marketing-campaign success and customer-satisfaction rewards to illustrate why real-world transition rules may be unknown.
What it assumes
Assumes familiarity with Markov decision processes and their state, action, reward, and transition framing.
What you can do after
Understand how an agent can learn action values from experience without knowing the environment’s transition dynamics.
How it connects
Connects model-free learning to Markov decision processes, dynamic programming, temporal-difference learning, Q-learning, and SARSA.

Also explained in

Compare how 3 books explain model-free method →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.