liveBook · Manning

Experience replay

As explained in Applied Reinforcement Learning

Experience replay is a reinforcement-learning technique that stores interaction transitions and trains on randomly sampled mini-batches to reduce temporal correlation.

Experience replay lets a reinforcement-learning agent learn from a mix of past interactions rather than only the latest sequence. It stores transitions in a replay buffer and samples random mini-batches for training. Mixing transitions reduces temporal correlation, helping stabilize gradient estimates in deep Q-learning. section 8.3 section 8.4

A typical replay cycle has four steps:

  • Record each transition as (state, action, reward, next state, done). The done flag indicates whether the episode ended. section 8.4
  • Add transitions to a fixed-capacity buffer. A first-in, first-out buffer discards the oldest transition when full. section 8.4
  • Once enough transitions have accumulated, sample a random mini-batch. A batch can mix experiences from different episodes. section 8.4 section 8.6
  • Use the sampled states, actions, rewards, next states, and termination flags to calculate training targets and update the Q-network. section 8.4

Replay addresses correlated experience; a target network addresses a different instability by keeping the parameters used for targets fixed between updates. Both mechanisms help make deep Q-learning trainable. section 8.3 section 8.6

Treatment optimization

In a simulated radiotherapy decision problem, a replay buffer holds up to 50,000 transitions and samples batches of 64. The agent waits until 1,000 transitions are available before training. A batch can combine decisions from different treatment episodes instead of repeating their original order. section 8.4

Listing 8.8 Replay buffer for storing transitions
class ReplayBuffer:
    def __init__(self, capacity: int):
        self.capacity = capacity  #A
        self.buffer = deque(maxlen=capacity)  #B
 
    def push(self, state, action, reward, next_state, done):
        self.buffer.append(
            (
                np.array(state, dtype=np.float32),
                int(action),
                float(reward),
                np.array(next_state, dtype=np.float32),
                bool(done),
            )
        )  #C
 
    def sample(self, batch_size: int):
        batch = random.sample(self.buffer, batch_size)  #D
        states, actions, rewards, next_states, dones =
        ➥zip(*batch)  #E
        return (
            np.stack(states),
            np.array(actions, dtype=np.int64),
            np.array(rewards, dtype=np.float32),
            np.stack(next_states),
            np.array(dones, dtype=np.bool_),
        )  #F
 
    def __len__(self):
        return len(self.buffer)  #G

The replay buffer uses a bounded deque, converts transition fields to consistent types, and samples batches for training. section 8.4

As a starting guideline, use a buffer of 10,000–100,000 transitions for simple problems and 1 million or more for complex domains. Begin training after collecting at least ten times the batch size. section 8.6

How Applied Reinforcement Learning covers it

How deep it goes
Explains transition storage, FIFO eviction, random sampling, training thresholds, and replay buffer sizing, with an implementation.
Examples
Illustrates replay in simulated radiotherapy treatment optimization with a 50,000-transition buffer and batches of 64.
What you can do after
Builds a replay buffer from scratch and uses sampled transitions in a Double DQN training step.
How it connects
Connects replay with deep Q-networks, temporal-difference targets, target networks, and Double DQN.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.