Chapter 6. Building your first deep neural network: introduction to backpropagation

“O Deep Thought computer,” he said, “the task we have designed you to perform is this. We want you to tell us...” he paused, “The Answer.”

Douglas Adams, The Hitchhiker’s Guide to the Galaxy

The streetlight problem

This toy problem considers how a network learns entire datasets

Consider yourself approaching a street corner in a foreign country. As you approach, you look up and realize that the street light is unfamiliar. How can you know when it’s safe to cross the street?

You can know when it’s safe to cross the street by interpreting the streetlight. But in this case, you don’t know how to interpret it. Which light combinations indicate when it’s time to walk? Which indicate when it’s time to stop? To solve this problem, you might sit at the street corner for a few minutes observing the correlation between each light combination and whether people around you choose to walk or stop. You take a seat and record the following pattern:

Preparing the data

Matrices and the matrix relationship

Creating a matrix or two in Python

Building a neural network

Learning the whole dataset

Full, batch, and stochastic gradient descent

Neural networks learn correlation

Up and down pressure

Edge case: Overfitting

Edge case: Conflicting pressure

Learning indirect correlation

Creating correlation

Stacking neural networks: A review