8 Reduce, reuse, and recycle your words: RNNs and LSTMs
This chapter covers
- Unrolling recursion, so you can understand how to use it for NLP
- Implementing word and character-based recurrent neural networks (RNNs) in PyTorch
- Identifying applications where RNNs are your best option
- Understanding backpropagation in time
- Making your RNN smarter with long- and short-term memory
Recurrent neural networks (RNNs) are a game changer for NLP. They have spawned an explosion of practical applications and advancements in deep learning and AI, including real-time transcription and translation on mobile phones, high-frequency algorithmic trading, and efficient code generation. RNNs recycle tokens, but why would you want to recycle and reuse your words? To build a more sustainable NLP pipeline, of course! Recurrence is just another word for recycling. An RNN uses recurrence to remember the tokens it has already read and reuse that understanding to predict the target variable. If you use RNNs to predict the next word, RNNs can generate, going on and on and on, until you tell them to stop. This sustainability or regenerative ability of RNNs is their superpower.