liveBook · Manning

Replay buffer

As explained in Applied Reinforcement Learning

A replay buffer is a storage structure that retains interaction records for later processing or learning updates.

A replay buffer stores an agent’s past interactions for later learning. In a Deep Q-Network (DQN), it acts like a rolling record of decisions and outcomes: instead of training immediately on each consecutive interaction, the agent samples from its stored experience. section 8.3

Consecutive interactions can be highly correlated. Training on them in order can make a network overfit to recent patterns and forget earlier ones. Randomly mixing transitions from different points in the agent’s history helps decorrelate training data and stabilize learning. section 8.3 section 8.4

A typical DQN replay buffer works as follows:

  • Store each transition as (state, action, reward, next state, done), where done records whether the episode ended. section 8.4
  • Keep a fixed capacity. When full, discard the oldest transition as a new one arrives. section 8.3 section 8.4
  • Draw a random mini-batch and separate its fields into arrays for a learning update. The transition fields support calculation of the temporal-difference target. section 8.4

Treatment optimization

In a radiotherapy training simulation, the buffer retains transitions from treatment decisions. One implementation uses a capacity of 50,000 and samples batches of 64, potentially mixing experiences from widely separated episodes. A training step waits until the buffer has at least 1,000 transitions before sampling. section 8.4

Listing 8.8 Replay buffer for storing transitions
class ReplayBuffer:
    def __init__(self, capacity: int):
        self.capacity = capacity  #A
        self.buffer = deque(maxlen=capacity)  #B
 
    def push(self, state, action, reward, next_state, done):
        self.buffer.append(
            (
                np.array(state, dtype=np.float32),
                int(action),
                float(reward),
                np.array(next_state, dtype=np.float32),
                bool(done),
            )
        )  #C
 
    def sample(self, batch_size: int):
        batch = random.sample(self.buffer, batch_size)  #D
        states, actions, rewards, next_states, dones =
        ➥zip(*batch)  #E
        return (
            np.stack(states),
            np.array(actions, dtype=np.int64),
            np.array(rewards, dtype=np.float32),
            np.stack(next_states),
            np.array(dones, dtype=np.bool_),
        )  #F
 
    def __len__(self):
        return len(self.buffer)  #G

The replay buffer listing shows a bounded Python deque, transition insertion, and random batch sampling. section 8.4

DQN is an off-policy method, so it can learn from transitions generated by an older policy and reuse them across updates. This contrasts with on-policy methods such as PPO, which use fresh data generated by the current policy. section 12.1

How Applied Reinforcement Learning covers it

How deep it goes
Explains transition storage, FIFO eviction, random sampling, batch arrays, and their stability rationale, with a Python implementation.
Examples
Uses a radiotherapy treatment simulation with a 50,000-transition buffer and batches of 64.
What you can do after
Build a fixed-capacity replay buffer from scratch and sample transition batches for DQN training.
How it connects
Connects experience replay with Deep Q-Networks, temporal-difference targets, off-policy learning, and catastrophic forgetting.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.