How deeply does Build a Robo-Advisor with Python (From Scratch) cover Q-learning?

Explains the state-action update and implements exploration, transitions, initialization, and episode generation.

Q-learning

As explained in Build a Robo-Advisor with Python (From Scratch)

Q-learning is a model-free reinforcement-learning algorithm that estimates the value of taking each action in each state and uses those estimates to derive a policy.

Q-learning estimates the value of each state-action pair and uses those estimates to derive an optimal policy. In goals-based financial planning, Q(W,t,a) evaluates an asset-allocation action at wealth W and time t. Unlike backward dynamic programming, it learns forward from t=0 by repeatedly running episodes and updating Q. section 6.2.2

The update combines current estimates with simulated experience:

Apply a learning rate α to the difference between the current Q value and the reward plus discounted best next-state Q value. section 6.2.2

Goals-based wealth planning

Initialize Q over wealth states, time points, and asset-allocation actions. Simulate the next wealth state, update the selected Q value, and repeat across 1,000,000 episodes to learn allocation values.

How Build a Robo-Advisor with Python (From Scratch) covers it

Cover of Build a Robo-Advisor with Python (From Scratch)

liveBook manning.com

Read Build a Robo-Advisor with Python (From Scratch) on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Evolutionary Deep Learning: Genetic algorithms and neural networksQ-learning is a model-free reinforcement-learning algorithm that learns action values by updating estimates of expected return for state-action pairs.
liveBook manning.com

Grokking AI Algorithms, Second EditionQ-learning is a reinforcement-learning algorithm that learns action values for states without requiring a model of the environment's transition dynamics.

Compare how 3 books explain Q-learning →

Related concepts

reinforcement learningkind of · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
epsilon-greedyuses · Epsilon-greedy is an action-selection strategy that chooses a random action with probability epsilon and otherwise selects the currently highest-valued action.
deep reinforcement learningcontrasts with · Deep reinforcement learning is reinforcement learning that uses deep neural networks to represent policies, value functions, or other decision-making components.

This explainer was generated by Manning's knowledge graph from the text of Build a Robo-Advisor with Python (From Scratch). The section links point to the parts of the book it draws on.