Experience replay lets a reinforcement-learning agent learn from a mix of past interactions rather than only the latest sequence. It stores transitions in a replay buffer and samples random mini-batches for training. Mixing transitions reduces temporal correlation, helping stabilize gradient estimates in deep Q-learning. section 8.3 section 8.4
A typical replay cycle has four steps:
- Record each transition as (state, action, reward, next state, done). The done flag indicates whether the episode ended. section 8.4
- Add transitions to a fixed-capacity buffer. A first-in, first-out buffer discards the oldest transition when full. section 8.4
- Once enough transitions have accumulated, sample a random mini-batch. A batch can mix experiences from different episodes. section 8.4 section 8.6
- Use the sampled states, actions, rewards, next states, and termination flags to calculate training targets and update the Q-network. section 8.4
Replay addresses correlated experience; a target network addresses a different instability by keeping the parameters used for targets fixed between updates. Both mechanisms help make deep Q-learning trainable. section 8.3 section 8.6
Treatment optimization
In a simulated radiotherapy decision problem, a replay buffer holds up to 50,000 transitions and samples batches of 64. The agent waits until 1,000 transitions are available before training. A batch can combine decisions from different treatment episodes instead of repeating their original order. section 8.4
Listing 8.8 Replay buffer for storing transitions
class ReplayBuffer:
def __init__(self, capacity: int):
self.capacity = capacity #A
self.buffer = deque(maxlen=capacity) #B
def push(self, state, action, reward, next_state, done):
self.buffer.append(
(
np.array(state, dtype=np.float32),
int(action),
float(reward),
np.array(next_state, dtype=np.float32),
bool(done),
)
) #C
def sample(self, batch_size: int):
batch = random.sample(self.buffer, batch_size) #D
states, actions, rewards, next_states, dones =
➥zip(*batch) #E
return (
np.stack(states),
np.array(actions, dtype=np.int64),
np.array(rewards, dtype=np.float32),
np.stack(next_states),
np.array(dones, dtype=np.bool_),
) #F
def __len__(self):
return len(self.buffer) #G
The replay buffer uses a bounded deque, converts transition fields to consistent types, and samples batches for training. section 8.4
As a starting guideline, use a buffer of 10,000–100,000 transitions for simple problems and 1 million or more for complex domains. Begin training after collecting at least ten times the batch size. section 8.6
