Q-learning

As explained in Evolutionary Deep Learning: Genetic algorithms and neural networks

Q-learning is a model-free reinforcement-learning algorithm that learns action values by updating estimates of expected return for state-action pairs.

Q-learning is a reinforcement-learning approach for choosing the next-best action from a known state. It assigns each state–action pair a quality value, then learns those values by trial and error: rewards reinforce useful choices, while failed attempts expose poor ones. A learned Q-table acts as a lookup table or policy for selecting actions. section 11.1.1 section 11.6

A small discrete environment makes the mechanism visible. In Frozen Lake, an agent moves up, down, left, or right across a 4-by-4 grid. Reaching the goal earns a reward; falling into a hole ends the episode with a negative reward. The agent must discover a reliable route while balancing random exploration against actions currently rated best. section 11.1.1

Figure 11.2 Frozen lake environment

Use the grid layout to connect states, available moves, terminal holes, and the rewarded goal before constructing the table. section 11.1.1

A table-based agent can be built in a short loop:

• Create the Gym environment, then read its action-space and observation-space sizes. section 11.1.1
• Initialize a zero-filled Q-table with one row per state and one column per action. section 11.1.1
• Choose either the highest-valued action or a random action according to epsilon exploration. section 11.1.1
• After each step, update the selected table value toward immediate reward plus the discounted best value of the next state. section 11.1.1
• Repeat episodes, decay epsilon, inspect average reward and the resulting table, then run greedy evaluation episodes. section 11.1.1

Frozen Lake Q-table

With 16 grid states and four movement actions, the initial Q-table contains zeros. Training for repeated episodes changes entries only after observed transitions and rewards. At evaluation time, choosing argmax(Q[state, :]) removes random exploration and follows the table’s preferred move at each state. section 11.1.1

Q-tables fit small, discrete state spaces but become impractical as environments grow more complex. Deep Q-learning, or DQN, replaces the table with a neural network trained using the Q-learning equation as a target. This extends the approach to environments such as Atari games, CartPole, and Lunar Lander, using stored experiences and replayed batches to update the model. section 11.2 section 11.5

Reward accumulation can show gradual improvement, but complex deep reinforcement learning still faces challenges such as multitask or generalized learning. Experiment with environments, episode counts, exploration schedules, and parameter settings rather than treating one run as a universal result. section 11.4.1 section 11.5

How Evolutionary Deep Learning: Genetic algorithms and neural networks covers it

How deep it goes

Walks through a runnable Q-table implementation, exploration, update rule, training loop, and evaluation before deferring complex environments to DQN.

Examples

Demonstrates a 4-by-4 Frozen Lake agent and relates deep Q-learning to CartPole, Atari, and Lunar Lander.

What it assumes

Teaches setup from Gym installation through a Q-table without stating formal reinforcement-learning prerequisites.

What you can do after

Build it from scratch as a Gym-based Q-table agent with epsilon exploration, reward updates, and greedy evaluation.

How it connects

Connects reinforcement learning with OpenAI Gym, epsilon exploration, deep Q-learning, neural networks, DQN, and NEAT.

liveBookmanning.com

Read Evolutionary Deep Learning: Genetic algorithms and neural networks on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionQ-learning is a reinforcement-learning algorithm that learns action values for states without requiring a model of the environment's transition dynamics.
• 

  liveBookmanning.com
  Build a Robo-Advisor with Python (From Scratch)Q-learning is a model-free reinforcement-learning algorithm that estimates the value of taking each action in each state and uses those estimates to derive a policy.

Compare how 3 books explain Q-learning →

Related concepts

• reinforcement learningkind of · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
• epsilon-greedyuses · Epsilon-greedy is an action-selection strategy that chooses a random action with probability epsilon and otherwise selects the currently highest-valued action.
• deep reinforcement learningcontrasts with · Deep reinforcement learning is reinforcement learning that uses deep neural networks to represent policies, value functions, or other decision-making components.

This explainer was generated by Manning's knowledge graph from the text of Evolutionary Deep Learning: Genetic algorithms and neural networks. The section links point to the parts of the book it draws on.