A discounted reward multiplies a reward’s contribution by a temporal discount factor, reducing the influence of rewards across time. In this NEAT LunarLander implementation, discounting evaluates genome fitness rather than directly changing agent actions. section 11.4
Each genome carries a discount value, inherits it during crossover, and mutates it within bounded limits. Recorded episode histories are replayed, convolved with discount powers, normalized, and compared with network outputs to produce reward error. section 11.4
Listing 11.19 EDL_11_4_NEAT_LunarLander.ipynb: Computing fitness
def compute_fitness(genome, net, episodes, min_reward, max_reward):
m = int(round(np.log(0.01) / np.log(genome.discount)))
discount_function = [genome.discount ** (m - i)
for I in range(m + 1)] #1
reward_error = []
for score, data in episodes: #2
# Compute normalized discounted reward.
dr = np.convolve(data[:, -1], discount_function)[m:]
dr = 2 * (dr–- min_reward) / (max_reward–- min_reward)–- 1.0
dr = np.clip(dr, -1.0, 1.0) #3
for row, dr in zip(data, dr): #4
observation = row[:8]
action = int(row[8])
output = net.activate(observation)
reward_error.append(float((output[action]–- dr)
** 2)) #5
return reward_error
Inspect the fitness implementation that constructs discount powers, computes normalized discounted rewards, and accumulates output error. section 11.4
