concept target network in category deep learning
appears as: target networks, target network, The target network, target network, A target network

This is an excerpt from Manning's book Deep Reinforcement Learning in Action.
This is not just a theoretical issue—it’s something that DeepMind observed in their own training. The solution they devised is to duplicate the Q-network into two copies, each with its own model parameters: the “regular” Q-network and a copy called the target network (symbolically denoted
-network, read “Q hat”). The target network is identical to the Q-network at the beginning, before any training, but its own parameters lag behind the regular Q-network in terms of how they’re updated.
Let’s run through the sequence of events again, with the target network in play (we’ll leave out the details of experience replay):
Figure 3.15. This is the general overview for Q-learning with a target network. It’s a fairly straightforward extension of the normal Q-learning algorithm, except that you have a second Q-network called the target network whose predicted Q values are used to backpropagate through and train the main Q-network. The target network’s parameters are not trained, but they are periodically synchronized with the Q-network’s parameters. The idea is that using the target network’s Q values to train the Q-network will improve the stability of the training.
![]()
Listing 3.8. DQN with experience replay and target network
from collections import deque epochs = 5000 losses = [] mem_size = 1000 batch_size = 200 replay = deque(maxlen=mem_size) max_moves = 50 h = 0 sync_freq = 500 #1 j=0 for i in range(epochs): game = Gridworld(size=4, mode='random') state1_ = game.board.render_np().reshape(1,64) + np.random.rand(1,64)/100.0 state1 = torch.from_numpy(state1_).float() status = 1 mov = 0 while(status == 1): j+=1 mov += 1 qval = model(state1) qval_ = qval.data.numpy() if (random.random() < epsilon): action_ = np.random.randint(0,4) else: action_ = np.argmax(qval_) action = action_set[action_] game.makeMove(action) state2_ = game.board.render_np().reshape(1,64) + np.random.rand(1,64)/100.0 state2 = torch.from_numpy(state2_).float() reward = game.reward() done = True if reward > 0 else False exp = (state1, action_, reward, state2, done) replay.append(exp) state1 = state2 if len(replay) > batch_size: minibatch = random.sample(replay, batch_size) state1_batch = torch.cat([s1 for (s1,a,r,s2,d) in minibatch]) action_batch = torch.Tensor([a for (s1,a,r,s2,d) in minibatch]) reward_batch = torch.Tensor([r for (s1,a,r,s2,d) in minibatch]) state2_batch = torch.cat([s2 for (s1,a,r,s2,d) in minibatch]) done_batch = torch.Tensor([d for (s1,a,r,s2,d) in minibatch]) Q1 = model(state1_batch) with torch.no_grad(): Q2 = model2(state2_batch) #2 Y = reward_batch + gamma * ((1-done_batch) * \ torch.max(Q2,dim=1)[0]) X = Q1.gather(dim=1,index=action_batch.long() \ .unsqueeze(dim=1)).squeeze() loss = loss_fn(X, Y.detach()) print(i, loss.item()) clear_output(wait=True) optimizer.zero_grad() loss.backward() losses.append(loss.item()) optimizer.step() if j % sync_freq == 0: #3 model2.load_state_dict(model.state_dict()) if reward != -1 or mov > max_moves: status = 0 mov = 0 losses = np.array(losses)