concept value - base method in category reinforcement learning

This is an excerpt from Manning's book Grokking Deep Reinforcement Learning.
In practice, both Adam and RMSprop are sensible choices for value-based deep reinforcement learning methods. I make extensive use of both in the chapters ahead. However, I do prefer RMSprop for value-based methods, as you’ll soon notice. RMSprop is stable and less sensitive to hyperparameters, and this is particularly important in value- based deep reinforcement learning.
In the past three chapters, we dived into the field of value-based DRL. We started with a simple approach, NFQ. Then, we made this technique more stable with the improvements presented in DQN and DDQN. Then, we made it more sample-efficient with Dueling DDQN and PER. Overall we have a pretty robust algorithm. But, just like with everything, value-based methods also have cons. First, they are sensitive to hyperparameters. This is well-known, but you should try it for yourself; go and change any hyperparameter, you can find more values that don’t work, than values that do. Second, value-based methods assume they interact with a Markovian environment, that the states contain all information required by the agent. This assumption dissipates as we move away from bootstrapping and value- based methods in general. Lastly, the combination of bootstrapping, off-policy learning, and function approximators are known conjointly as ‘the deadly triad.’ While the ‘deadly triad’ is known to produce divergence, researchers still don’t know exactly how to prevent it.
Now, by no means, I’m saying that value-based methods are inferior to the methods we survey in future chapters. Those methods have issues of their own, too. The fundamental takeaway is to know that value-based deep reinforcement learning methods are well-known to diverge, and that is their weakness. How to fix it is still a research question, but sound practical advice is to use target networks, replay buffers, double learning, sufficiently small learning rates (but not too small), and maybe a little bit of patience. I’m sorry about that; I don’t make the rules.