2 Deep learning and neural networks

“If the human brain were so simple that we could understand it, we would be so simple that we couldn’t.”

-- George Edgin Pugh

In the last chapter we discussed the computer vision pipeline components: 1) input image, 2) preprocessing, 3) extracting features, and 4) learning algorithm (classifier). We also discussed that in traditional ML algorithms, we manually extract features that produces a vector of features to be classified by the learning algorithm. Whereas, in deep learning, neural networks acts as the feature extractor + classifier. It automatically recognizes patterns and extracts features from the image and classifies them into labels.

In this chapter, we will take a short pause from the computer vision context to open the “deep learning algorithm” box from the figure above. We will dive deeper into how neural networks learn features and make predictions.  Then, in the next chapter, we will come back to computer vision applications with one of the most popular deep learning architectures, Convolutional Neural Networks (CNNs).

The high-level layout of this chapter will be as follows:

2.1   The Perceptron intuition

2.1.1   What is a perceptron?

2.1.2   How does the perceptron learn?

2.1.3   Is one neuron enough to solve complex problems?

2.2   Multi-Layer Perceptron (MLP)

2.2.1   Multi-Layer Perceptron Architecture

2.2.2   What are the Hidden Layers?

2.2.3   How many layers and how many nodes in each layer?

2.2.4   MLP Takeaways

2.3   Activation functions

2.3.1   Linear Transfer Function

2.3.2   Heaviside Step Function (Binary classifier)

2.3.3   Sigmoid/Logistic function

2.3.4   Softmax Function

2.3.5   Hyperbolic Tangent Function (tanh)

2.3.6   Rectified Linear Unit (ReLU)