7 Function approximation: How neural networks model the world

 

This chapter covers

  • Expressing real-world problems as mathematical functions
  • Understanding the building blocks of a neural network
  • Approximating functions via neural networks

Computing to date has been dominated by the von Neumann architecture in which the processor and the program are separate. The program sits in memory and is fetched and executed by the processor. The advantage of this approach is that different programs solving unrelated problems can be loaded into memory, and the same processor can execute them. But neural networks have a fundamentally different architecture. There are no separate processors and programs; instead, there is a single entity called, well, the neural network, which can run on dedicated hardware or a Von Neumann computer. In this chapter, we discuss this paradigm in detail.

NOTE

The complete PyTorch code for this chapter is available at http://mng.bz/K4zj in the form of fully functional and executable Jupyter notebooks.

7.1 Neural networks: A 10,000-foot view

7.2 Expressing real-world problems: Target functions

7.2.1 Logical functions in real-world problems

7.2.2 Classifier functions in real-world problems

7.2.3 General functions in real-world problems

7.3 The basic building block or neuron: The perceptron

7.3.1 The Heaviside step function

7.3.2 Hyperplanes

7.3.3 Perceptrons and classification

7.3.4 Modeling common logic gates with perceptrons

7.4 Toward more expressive power: Multilayer perceptrons (MLPs)

7.4.1 MLP for logical XOR

7.5 Layered networks of perceptrons: MLPs or neural networks

7.5.1 Layering

7.5.2 Modeling logical functions with MLPs

7.5.3 Cybenko’s universal approximation theorem

7.5.4 MLPs for polygonal decision boundaries

Summary