5 Word brain: Neural networks

 

This chapter covers

  • Building a base layer for your neural networks
  • Training neural networks with backpropagation
  • Implementing a basic neural network in Python
  • Implementing a scalable neural network in PyTorch
  • Stacking network layers for better data representation
  • Tuning up your neural network for better performance

When you read word brain in the title of this chapter, the neurons in your brain likely started firing, trying to remind you where you’d heard something like that before. And now that you read the word heard, your neurons might be connecting the words in the title to the part of your brain that processes the sound of words. And maybe, the neurons in your auditory cortex are starting to connect the phrase word brain to common phrases that rhyme with it, such as bird brain.

Even if our brains didn’t predict your brain very well, you’re about to build a small brain yourself that can process a single word and predict something about what it means. This word brain you are about to build will be a lot better than our collective human brains, at least for some particularly hard NLP tasks. A neural net can even make predictions about meaning when the word it is processing is a person’s name that doesn’t seem to mean anything at all to a human.

5.1 Why neural networks?

5.1.1 Neural networks for words

5.1.2 Neurons as feature engineers

5.1.3 Biological neurons

5.1.4 Perceptron

5.1.5 A Python perceptron

5.2 An example logistic neuron

5.2.1 The logistics of clickbait

5.2.2 Sex education

5.2.3 Pronouns, gender, and sex

5.2.4 Sex logistics

5.2.5 A sleek, new PyTorch neuron

5.3 Skiing down the error slope

5.3.1 Off the chair lift, onto the slope: Gradient descent and local minima

5.3.2 Shaking things up: Stochastic gradient descent

5.4 Test yourself

Summary