Chapter 11. Neural networks that understand language: king – man + woman == ?

 

In this chapter

  • Natural language processing (NLP)
  • Supervised NLP
  • Capturing word correlation in input data
  • Intro to an embedding layer
  • Neural architecture
  • Comparing word embeddings
  • Filling in the blank
  • Meaning is derived from loss
  • Word analogies

“Man is a slow, sloppy, and brilliant thinker; computers are fast, accurate, and stupid.”

John Pfeiffer, in Fortune, 1961

What does it mean to understand language?

What kinds of predictions do people make about language?

Up until now, we’ve been using neural networks to model image data. But neural networks can be used to understand a much wider variety of datasets. Exploring new datasets also teaches us a lot about neural networks in general, because different datasets often justify different styles of neural network training according to the challenges hidden in the data.

We’ll begin this chapter by exploring a much older field that overlaps deep learning: natural language processing (NLP). This field is dedicated exclusively to the automated understanding of human language (previously not using deep learning). We’ll discuss the basics of deep learning’s approach to this field.

Natural language processing (NLP)

NLP is divided into a collection of tasks or challenges

Perhaps the best way to quickly get to know NLP is to consider a few of the many challenges the NLP community seeks to solve. Here are a few types of classification problem that are common to NLP:

Supervised NLP

IMDB movie reviews dataset

Capturing word correlation in input data

Predicting movie reviews

Intro to an embedding layer

Interpreting the output

Neural architecture

Comparing word embeddings

What is the meaning of a neuron?

Filling in the blank

Meaning is derived from loss

King – Man + Woman ~= Queen

Word analogies

Summary