A replay buffer stores an agent’s past interactions for later learning. In a Deep Q-Network (DQN), it acts like a rolling record of decisions and outcomes: instead of training immediately on each consecutive interaction, the agent samples from its stored experience. section 8.3
Consecutive interactions can be highly correlated. Training on them in order can make a network overfit to recent patterns and forget earlier ones. Randomly mixing transitions from different points in the agent’s history helps decorrelate training data and stabilize learning. section 8.3 section 8.4
A typical DQN replay buffer works as follows:
- Store each transition as (state, action, reward, next state, done), where done records whether the episode ended. section 8.4
- Keep a fixed capacity. When full, discard the oldest transition as a new one arrives. section 8.3 section 8.4
- Draw a random mini-batch and separate its fields into arrays for a learning update. The transition fields support calculation of the temporal-difference target. section 8.4
Treatment optimization
In a radiotherapy training simulation, the buffer retains transitions from treatment decisions. One implementation uses a capacity of 50,000 and samples batches of 64, potentially mixing experiences from widely separated episodes. A training step waits until the buffer has at least 1,000 transitions before sampling. section 8.4
Listing 8.8 Replay buffer for storing transitions
class ReplayBuffer:
def __init__(self, capacity: int):
self.capacity = capacity #A
self.buffer = deque(maxlen=capacity) #B
def push(self, state, action, reward, next_state, done):
self.buffer.append(
(
np.array(state, dtype=np.float32),
int(action),
float(reward),
np.array(next_state, dtype=np.float32),
bool(done),
)
) #C
def sample(self, batch_size: int):
batch = random.sample(self.buffer, batch_size) #D
states, actions, rewards, next_states, dones =
➥zip(*batch) #E
return (
np.stack(states),
np.array(actions, dtype=np.int64),
np.array(rewards, dtype=np.float32),
np.stack(next_states),
np.array(dones, dtype=np.bool_),
) #F
def __len__(self):
return len(self.buffer) #G
The replay buffer listing shows a bounded Python deque, transition insertion, and random batch sampling. section 8.4
DQN is an off-policy method, so it can learn from transitions generated by an older policy and reuse them across updates. This contrasts with on-policy methods such as PPO, which use fresh data generated by the current policy. section 12.1
