Epsilon-greedy balances exploration and exploitation during reinforcement learning. At each state, it selects a random action with probability ε and the action with the highest expected cumulative reward Q(W,t,a) with probability 1−ε. Pure exploitation can miss better policies, while pure exploration wastes existing knowledge. section 6.2.2
Selecting and updating an action
With ε = 0.5, the agent explores half the time on average. Otherwise it selects an action maximizing Q; ties are broken randomly. After transitioning to the next wealth state, it updates the selected value using α, γ, and the maximum next-state Q value. section 6.2.2
Listing 6.5 Updating \(Q\)
def Update_Q(w_idx, t, Q):
if np.random.uniform() < epsilon: #1
a = np.random.randint(nA)
else:
A_array = Q[w_idx, t, :]
a = np.where(A_array == A_array.max())[0]
if (len(a)>1): #2
a = np.random.choice(a)
W_tplus1_idx = transition(W[w_idx,t], t, a)
Q[w_idx, t, a] += alpha*(gamma*Q[W_tplus1_idx, t+1, :].max()
- Q[w_idx, t, a]) #3
return W_tplus1_idx, t+1, Q
Inspect the Python implementation for ε-based exploration, greedy selection, random tie-breaking, state transitions, and Q-value updates. section 6.2.2
Listing 6.6 Generating episodes
from tqdm import tqdm
n_episodes = 1000000
epsilon = 0.5
alpha = 0.1
gamma = 1.0
Q = np.zeros((nW, T+1, nA))
for j in range(nW):
if W[j, T]>G:
Q[j, T, :] = 1
for i in tqdm(range(n_episodes), desc="Running..."):
W_idx = 0
t = 0
while t<=(T-1):
W_idx, t, Q = Update_Q(W_idx, t, Q)
Review the episode-generation loop, including ε, α, γ, Q initialization, terminal values, and repeated learning paths. section 6.2.2
