Chapter 7: Function approximation: How neural networks model the world

Introduction

This chapter introduces neural networks as a computing paradigm that represents real-world problems as functions and approximates those functions with network building blocks.

• Neural network computing paradigm: Describe how neural networks differ architecturally from traditional von Neumann computing: rather than a separate processor executing a separately stored program, a neural network is a single entity that may run on dedicated hardware or on a von Neumann computer.
  Why it matters: This contrast frames why neural networks are treated as models whose structure itself performs the computation.
• Function approximation roadmap: State that the chapter's program is to express real-world problems as mathematical functions, understand neural-network building blocks, and approximate functions with neural networks.
  Why it matters: This roadmap connects the chapter's examples of functions, perceptrons, and multilayer networks into one purpose.

Key terms: von Neumann architecture, processor, program, neural network, dedicated hardware, real-world problems, mathematical functions, building blocks, function approximation, neural networks

7.1 Neural networks: A 10,000-foot view

This section presents neural networks as a practical, unified way to approximate unknown real-world target functions through architecture choice and training.

• Neural networks as function approximators: Explain why neural networks can model many different real-world intelligent tasks as approximations to target functions, even when those target functions are arbitrarily complicated and unknown.
  Why it matters: This function-approximation view explains why one general neural-network framework can be applied to many otherwise unrelated problems.
• Expressive power and network design: Describe expressive power as the variety and complexity of functions a network can represent, and explain how neuron count and connectivity affect it. Distinguish architecture selection from choosing connection-weight parameter values.
  Why it matters: Matching expressive power to problem complexity is central to designing a network that can represent the needed behavior.
• Supervised training and inferencing: Trace the supervised-neural-network workflow from manually labeled input-output pairs through weight training to inferencing on previously unseen inputs. State that inferencing accuracy is not guaranteed, but is probabilistic and depends on sufficient expressive power, proper training, and adequate data.
  Why it matters: This is the basic lifecycle by which a supervised network is made useful on new real-world inputs.
• Alternatives to manual labeling: Distinguish supervised learning from unsupervised, semi-supervised, and self-supervised neural networks in terms of how ground-truth labels are supplied. State the section's practical trade-off that manual labeling is costly, while unsupervised and semi-supervised methods are less mature and harder to bring to desired accuracy levels.
  Why it matters: The source and cost of labels strongly affects which learning approach is feasible for a problem.
• Closed-form functions unnecessary: Explain why neural-network practice does not require a closed-form expression for either the unknown target function or the network's overall modeling function, and why this makes the technology practical.
  Why it matters: Neural networks are useful precisely because real tasks can be learned from sample input–ground-truth pairs without deriving an intractable global formula.

Key terms: target function, modeling function, function approximation, neural network, expressive power, architecture, neurons, connections, parameters, weights

7.2 Expressing real-world problems: Target functions

This section frames real-world problems as numeric target functions that map input variables to output variables and distinguishes continuous from categorical quantities.

• Target-function problem formulation: Explain how a real-world problem can be represented by a target function that takes quantifiable input variables and produces numeric output variables for a specific input instance. State the usual notation x for inputs, y for outputs, and y = f(x) for the target function.
  Why it matters: Expressing a problem as an input-to-output function is the starting point for modeling it with a neural network.
• Continuous and categorical quantities: Distinguish continuous variables, which can take infinitely many real values in a range, from categorical variables, which take one of a finite set of category values. Describe a binary variable as the special categorical case with two categories, such as sell versus do not sell.
  Why it matters: The kind of numeric quantity used for inputs or outputs helps define the target function being modeled.
• Target-function families: State that the section divides target functions into three families: logical functions, classifier functions, and general functions.
  Why it matters: These three families organize the target-function examples developed in the following subsections.

Key terms: target function, input variables, output variables, x, y, y = f(x), continuous variables, categorical variables, binary variable, categories

7.2.1 Logical functions in real-world problems

This section shows how real-world binary decisions can be formed by combining outputs from separate detectors with logical functions.

• Binary logical-function framing: Explain that logical functions take binary inputs and produce binary outputs, where 0 and 1 can represent no/yes or do not fire/fire. Describe how a logical decision machine can sit on top of separate machines that each detect or decide one input condition.
  Why it matters: This framing shows how larger decision systems can be built by combining simpler detectors.
• Core logical operator behavior: Describe the decision behavior of OR, AND, NOT, and XOR for binary conditions, using the section’s examples. Distinguish XOR, which is 1 when exactly one condition is true, from the social-media recommendation rule, which uses NOT XOR to favor matching music preferences.
  Why it matters: These operators are the basic rules used to turn multiple detected conditions into a binary action or recommendation.
• Multi-input and threshold triggers: Describe multi-input OR and AND decisions and an m-out-of-n trigger. Explain why a face detector can declare a face after detecting only a specified number of parts, rather than all parts, to remain robust to occlusion.
  Why it matters: Threshold-style decisions let a system act reliably when some useful signals are absent or blocked.

Key terms: logical functions, binary variables, binary decision, separate machines, logical OR, logical AND, logical NOT, logical XOR, NOT of the exclusive OR, multi-input logical OR

7.2.2 Classifier functions in real-world problems

This section presents classifiers as functions that partition a feature space into class regions by estimating decision boundaries from labeled training data.

• Classifier function representation: Describe a classifier as a function that maps an input, whether continuous or categorical, to a categorical class output. Explain how an image or other input can be represented as an input vector, with each possible input forming a point in a feature space.
  Why it matters: This function-and-geometry view is the basis for understanding what a neural-network classifier is trying to construct.
• Decision-boundary geometry: Explain that a classifier partitions feature space into regions associated with different classes, and that the separating curve, surface, or hypersurface is its decision boundary. State that estimating this boundary is effectively the same as building the classifier.
  Why it matters: Decision boundaries provide the geometric language for judging whether a model can separate the classes required by a problem.
• Training-data representativeness: Explain that training data consists of labeled input and ground-truth-output pairs, and that training optimizes a decision boundary to classify those sampled points well. Describe why this boundary will be useful only when the samples from each class reasonably reflect and span that class's true region in input space.
  Why it matters: A model can fit its provided examples yet learn a poor boundary when the training samples do not represent the real class distribution.
• Sign-based boundary classification: State that an equation of the form ϕ(x)=0 defines a decision surface and explain that the sign of ϕ(x) determines which side of that surface an input point lies on. For a linear boundary ϕ(x)=wᵀx+b, distinguish negative, zero, and positive values as the two sides and the boundary itself.
  Why it matters: The sign of a boundary function turns a geometric separator into a concrete mathematical classification rule.

Key terms: classifier, categorical output, class, input space, feature space, input vector, decision boundary, partition, class region, hypersurface

7.2.3 General functions in real-world problems

This section shows that real-world target functions may need continuous outputs, either to estimate a quantity such as driving speed or to express an event’s probability.

• Continuous target outputs: Describe why some real-world problems require a continuous output rather than a categorical output, using the self-driving vehicle speed example. State that inputs such as the road’s speed limit and neighboring vehicles’ speeds can be used to estimate how fast the vehicle should travel.
  Why it matters: Neural networks are not limited to choosing categories; they can model quantities that vary continuously.
• Discriminative and generative outputs: Distinguish a discriminative face-classification function, which outputs 0 for not a face and 1 for a face, from a generative function that outputs the probability that an image contains a face.
  Why it matters: The form of a model’s output determines whether it makes a hard category decision or expresses uncertainty as a probability.

Key terms: continuous output variable, categorical output variable, discriminative, generative, face classification function, probability

7.3 The basic building block or neuron: The perceptron

This introduction motivates neural networks as a general framework for approximating unknown real-world functions from sample input-output pairs using simple neurons.

• Neural networks as function models: Explain why neural networks offer a repeatable framework for modeling many different unknown real-world target functions. State that they can be estimated from sample input-output pairs without requiring the target function in closed form.
  Why it matters: This is the central reason neural networks can be used across problems whose underlying rules are unknown.
• Neuron composition and expressive power: Describe the neuron as the simple building block of a neural network and explain that connecting many such building blocks allows the complete network to model highly complicated functions.
  Why it matters: The contrast between simple neurons and an expressive full network motivates the later construction of multilayer networks.

Key terms: neural networks, target function, closed form, sample input-output pairs, modeling, neuron, building block, neural network, expressive power

7.3.1 The Heaviside step function

This section defines the Heaviside step function as a binary threshold function and relates its equation, algorithm, and graph.

• Heaviside step function behavior: Describe the Heaviside step function as a function that outputs 0 for negative arguments and 1 for positive arguments, and relate this rule to its algorithmic and graphical forms.
  Why it matters: This binary threshold behavior is the activation rule used to understand the perceptron.

Key terms: Heaviside step function, step function, negative arguments, positive arguments

7.3.2 Hyperplanes

This section explains how a hyperplane is defined by a weighted input expression, why its weight vector is normal to it, and how its sign partitions space for classification.

• Hyperplane equation and normal: Describe a hyperplane as the set of input-space points satisfying x^T w + b = 0 for fixed vector w and scalar b, and explain why w is normal to the plane.
  Why it matters: The normal vector and offset specify the geometric boundary that a perceptron uses to separate inputs.
• Sign-based space partition: Explain how the sign of x^T w + b partitions input space into two sides of a hyperplane and lets that hyperplane serve as a decision boundary.
  Why it matters: A classifier can make a binary decision by determining on which side of its boundary an input point lies.

Key terms: hyperplane, x^T w + b = 0, normal, weight vector, bias, orthogonal, decision boundary, hypersurface, x^T w + b, positive

7.3.3 Perceptrons and classification

This section explains how a perceptron uses a Heaviside step applied to a hyperplane to classify points, why its parameters must be trained from labeled examples, and why one perceptron cannot represent nonplanar class regions.

• Perceptron decision boundary: Explain how a perceptron combines a hyperplane expression with the Heaviside step function to act as a linear classifier. Describe how it assigns points on opposite sides of its hyperplane decision boundary to outputs 0 and 1.
  Why it matters: This is the basic connection between a neuron’s mathematical formula and binary classification.
• Trained separator parameters: Describe why labeled training samples are needed to choose the perceptron's weight and bias values, and distinguish parameter values that partition the training points well from ones that partition them poorly. State that training determines good values of these parameters.
  Why it matters: A classifier’s boundary is not known in advance; its parameters must be selected from labeled data.
• Single-perceptron limitation: Explain that one perceptron can only partition the input space with one planar decision surface, so it cannot classify a class region that lies between two planes while the other class occupies the remaining space. State that more complex boundaries require multiple perceptrons.
  Why it matters: This limitation motivates multilayer networks, which can combine simple separators into more expressive classifiers.

Key terms: perceptron, Heaviside step function, hyperplane, linear classifier, decision boundary, training data, manually labeled classes, weights, bias, planar separator

7.3.4 Modeling common logic gates with perceptrons

This section shows how single perceptrons implement the binary AND, OR, and NOT logic gates by using weighted sums, biases, and thresholding.

• Perceptron logic-gate construction: Describe how a perceptron can model AND, OR, and NOT logical functions by choosing weights and a bias so its step-function output matches the gate's binary truth table.
  Why it matters: This is the basic example of how learned-looking neural-network components can represent concrete input-output functions.
• AND and OR thresholds: State how weights of 1 and different biases implement two-input AND and OR: AND uses the threshold condition x0 + x1 >= 1.5, whereas OR uses x0 + x1 >= 0.5.
  Why it matters: These examples make clear that changing only the bias can change a perceptron's logical behavior.
• Hyperplane gate geometry: Explain that a perceptron's threshold equation forms a line in the two-input plane that partitions input points into half-planes with output zero or one, placing the logic-gate input combinations on the appropriate side.
  Why it matters: The geometric view connects a perceptron's algebraic threshold to its role as a classifier and decision boundary.

Key terms: perceptron, weighted edges, weights, bias, Heaviside step function, logical AND, logical OR, threshold, ϕ(x0 + x1 - 1.5), hyperplane

7.4 Toward more expressive power: Multilayer perceptrons (MLPs)

7.4.1 MLP for logical XOR

This subsection uses XOR to show why a single perceptron is insufficient and why multiple perceptrons are needed for more expressive models.

• XOR input partition: Describe how the four XOR input points are partitioned by output: (0,0) and (1,1) map to 0, while (0,1) and (1,0) map to 1.
  Why it matters: The XOR arrangement makes the geometric limitation of a single linear classifier visible.
• Single-perceptron XOR limitation: Explain why one perceptron cannot model XOR: no single straight-line decision boundary can place both output-1 points on one side and both output-0 points on the other. State that multiple perceptrons can model XOR instead.
  Why it matters: This is a direct example of why neural networks gain expressive power by combining perceptrons.

Key terms: logical XOR function, input points, output value, perceptron, linear decision boundary, straight line, multiple perceptrons

7.5 Layered networks of perceptrons: MLPs or neural networks

This section introduces neural networks as organized networks of multiple perceptrons needed to solve problems beyond the capability of a single perceptron.

• Need for perceptron networks: Explain why a single perceptron is insufficient for many practical problems and why connecting multiple perceptrons forms a neural network.
  Why it matters: This establishes why multilayer neural networks are needed rather than relying on one perceptron.

Key terms: perceptron, neural network, network of connected perceptrons, XOR

7.5.1 Layering

This subsection defines the layered connection pattern of multilayer perceptrons, hidden layers, layer-specific parameters, and why multiple hidden layers make a network deep.

• Layered MLP connectivity: Describe how perceptrons are organized into numbered layers in an MLP, with each perceptron's output feeding only perceptrons in the immediately following layer. Explain that connections skipping layers or going to other layers are not allowed in this organization.
  Why it matters: This restricted flow makes an MLP a manageable feed-forward network and establishes the structure used for training.
• Hidden layer role: Explain that the outputs of every layer except the final layer are not directly visible as the network's output, and that these intermediate layers are called hidden layers. Identify layer 0 in the XOR example as a hidden layer.
  Why it matters: Hidden layers are the intermediate computational stages that let an MLP build an output through successive transformations.
• Layer-owned parameters and depth: State that every weight and bias belongs to exactly one layer and that the book denotes its layer index with a parenthesized superscript. Explain that an MLP with two or more hidden layers may be called a deep neural network, which gives deep learning its name.
  Why it matters: Layer-specific parameter notation and the count of hidden layers are used to describe and train deeper network architectures precisely.

Key terms: layering, multilayer perceptron, MLP, layer, input, output, hidden layers, output layer, layer 0, weight

7.5.2 Modeling logical functions with MLPs

This section shows how to construct an MLP for any logical function by implementing the truth-table rows that output 1 and OR-ing them together.

• Truth-table MLP construction: Describe how to model an arbitrary logical function with an MLP from its truth table. Explain that each row whose output is 1 becomes an AND term of input variables or their complements, and the MLP ORs those term-perceptron outputs into the final result.
  Why it matters: This gives a constructive reason that layered perceptrons can represent logical functions of any number of variables.
• XOR row decomposition: Explain the two-input XOR example as the expression x̄0x1 + x0x̄1. State that its two output-1 rows are detected separately and then OR-ed, producing an MLP equivalent to the earlier XOR network.
  Why it matters: XOR makes the general truth-table construction concrete and illustrates why multiple perceptron layers are useful.

Key terms: truth table, MLP, perceptron, logical AND, logical OR, complements, logical XOR, x̄0x1 + x0x̄1, AND, OR

7.5.3 Cybenko’s universal approximation theorem

This section explains Cybenko’s universal approximation idea by showing how MLPs build one- and two-dimensional towers whose sums approximate continuous functions.

• Universal approximation by towers: Explain how a continuous function on an interval can be approximated by a sum of vertical rectangular towers, and connect this construction to Cybenko’s theorem for single-hidden-layer MLPs.
  Why it matters: This provides the fundamental reason neural networks can represent complicated continuous relationships.
• Single-layer practicality limitation: Describe the practical limitation of Cybenko’s theorem: although one hidden layer can model any continuous function, a complicated function may require an arbitrarily and impractically large number of perceptrons; explain why extra layers are therefore useful in practice.
  Why it matters: Universal approximation is a theoretical guarantee, not a claim that shallow networks are an efficient practical design.
• One-dimensional tower construction: Trace how perceptron step functions form a one-dimensional tower: use biases to shift steps, a negative weight to flip a step, and a logical AND of a shifted step and flipped shifted step to create a bounded tower.
  Why it matters: This concrete construction shows how simple perceptrons can be assembled into the building blocks of function approximation.
• Two-dimensional tower construction: Describe how MLPs construct a two-dimensional tower by forming a wave along each coordinate axis from paired shifted and flipped steps, then logically ANDing the two axis-specific waves.
  Why it matters: The two-dimensional construction extends the tower idea from curves to surfaces and shows how MLPs define bounded regions in input space.

Key terms: Cybenko’s theorem, continuous function, towers, single hidden layer, MLP, perceptrons, expressive power, decision boundary, step function, bias

7.5.4 MLPs for polygonal decision boundaries

This section shows how an MLP combines perceptrons for linear half-planes with an AND operation to classify a rectangular region, extending the idea to polygonal approximations of planar shapes.

• Rectangular decision-region construction: Describe how an MLP identifies points inside the rectangle defined by x0 ≥ -5, x0 ≤ 5, x1 ≥ -2, and x1 ≤ 2. Explain that four perceptrons test the four half-plane conditions and a logical AND makes the network fire only when every condition holds.
  Why it matters: This illustrates how networks turn simple linear boundary tests into a classifier for a bounded region.
• Polygonal approximation of planar shapes: Explain why the rectangular construction generalizes to arbitrary planar shapes: shapes on a plane can be approximated by polygons, and sufficiently many perceptrons can depict such shapes to arbitrary accuracy.
  Why it matters: This connects simple straight-line tests to the expressive ability of MLP classifiers for more complicated two-dimensional regions.

Key terms: decision boundary, feature space, half-plane, perceptron, logical AND, MLP, polygons, perceptrons, arbitrary level of accuracy, planar shapes

7.6 Summary

This summary explains how neural networks use layered perceptrons to approximate real-world functions and why successful training depends on representative labeled data.