10 Sample-efficient value-based methods
In this chapter:
- You implement a deep neural network architecture that exploits some of the nuances that exist in value-based deep reinforcement learning methods.
- You create a replay buffer that prioritizes experiences by how surprising they are.
- You build an agent that trains to a near-optimal policy in fewer number of episodes than all previous value-based deep reinforcement learning agents.
Intelligence is based on how efficient a species became at doing the things they need to survive.
— Charles Darwin, English naturalist, geologist, and biologist Best known for his contributions to the science of evolution.
In the previous chapter, we improved on NFQ with the implementation of DQN and DDQN. In this chapter, we continue on this line of improvements to previous algorithms by presenting two additional techniques for improving value-based deep reinforcement learning methods. This time, though, the improvements are not so much about stability, although that could easily be a by-product. But more accurately, the techniques presented in this chapter improve the sample-efficiency of DQN, and other value-based DRL methods.
First, we introduce a functional neural network architecture that splits the Q-function representation into two streams. One stream approximates the V-function, and the other stream approximates the A-function. V-functions are per-state values, while A-functions express the distance of each action from their V-functions.