A Q-table stores an agent’s estimated action-quality values. Each row represents a discrete state, and each column represents an available action in that state. The table acts as a compact lookup structure for choosing actions during reinforcement learning. section 11.1.1
Build and use a basic Q-table by:
- Creating a zero-filled array with shape (state_size, action_size), using the environment’s discrete state and action counts. section 11.1.1
- Selecting the action with the highest value for the current state, while sometimes sampling a random action to preserve exploration. section 11.1.1
- Updating the observed state-action entry with the reward, discounted value of the next state, and learning rate. section 11.1.1
- Repeating this process across episodes and steps, then gradually reducing epsilon so behavior shifts from exploration toward exploitation. section 11.1.1
FrozenLake implementation
For FrozenLake, initialize `Q = np.zeros((state_size, action_size))`. A policy can return `np.argmax(Q[state, :])` when exploiting or `env.action_space.sample()` when exploring. After an action produces `reward` and `new_state`, update `Q[state, action]` toward `reward + gamma * np.max(Q[new_state, :])`. Train for configured episodes and maximum steps, then run evaluation episodes using the highest-valued action. section 11.1.1
Listing 11.3 EDL_11_1_FrozenLake.ipynb: Building the Q-table
Q = np.zeros((state_size, action_size)) #1 print(Q) #========== printed Q table [[0. 0. 0. 0.] #2 [0. 0. 0. 0.] #3 [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.]]
Inspect the zero-initialized table and its state-by-action layout. section 11.1.1
Listing 11.6 EDL_11_1_FrozenLake.ipynb: The learn function
def learn(reward, state, action, new_state): Q[state, action] = Q[state, action] + learning_rate * (reward + gamma * np.max(Q[new_state, :]) – Q[state, action]) #1
Inspect the compact implementation of the Q-value update. section 11.1.1
Tabular Q-learning is practical when states and actions are discrete and manageable. As spaces become more complex, deep Q-learning replaces the table with a neural network; the table therefore marks a simpler alternative rather than a universal representation. section 11.2 section 11.6
