concept vanilla RNN in category deep learning

appears as: villa RNN
Grokking Deep Learning

This is an excerpt from Manning's book Grokking Deep Learning.

One last possible critique is about h. Clearly it’s still prone to vanishing and exploding gradients, because it’s basically being used the same as the vanilla RNN. First, because the h vector is always created using a combination of vectors that are squished with tanh and sigmoid, exploding gradients aren’t really a problem—only vanishing gradients. But this ends up being OK because h is conditioned on c, which can carry long-range information: the kind of information vanishing gradients can’t learn to carry. Thus, all long-range information is transported using c, and h is only a localized interpretation of c, useful for making an output prediction and constructing gate activations at the following timestep. In short, c can learn to transport information over long distances, so it doesn’t matter if h can’t.