Chapter 3: Classifiers and vector calculus

Introduction

This chapter introduces error minimization for classifiers by using gradients to adjust model weights and biases toward desired outputs.

• Gradient-based error minimization: Explain that training adjusts a model’s weights and biases so its outputs become closer to the desired outputs on one or more training examples, and that this chapter studies gradients as the means to do this.
  Why it matters: Gradient-based error minimization is the central idea behind training many machine learning and deep learning models.

Key terms: error minimization, training examples, weights, biases, gradients

3.1 Geometrical view of image classification

Introduces image classification through a car-versus-giraffe binary classifier and raises the need to represent the input image.

• Binary image classification setup: Describe the example classification task as deciding whether an input image contains a car or a giraffe, and state that a classifier with two possible classes is a binary classifier.
  Why it matters: This simple two-class example provides the concrete setting for discussing how classifiers represent inputs and make decisions.
• Input representation question: State that, before constructing the car-versus-giraffe classifier, one must decide how to represent the input image.
  Why it matters: A classifier can operate only on a usable representation of the image it receives.

Key terms: image classification, classifier, binary classifier, classes, input, input representation

3.1.1 Input representation

This section represents a fixed-size image as a pixel-value matrix and rasterizes that matrix into a high-dimensional vector for geometric machine-learning analysis.

• Image pixel matrix representation: Describe how a visual-scene image is represented as a two-dimensional matrix of pixels, including what a pixel value denotes and why images are scaled to a fixed size.
  Why it matters: Representing an image numerically is the first step toward giving it to a machine-learning model.
• Rasterized image vectors: Explain rasterization as converting an image matrix into a vector by reading entries left to right and top to bottom, while preserving all of the original pixel values.
  Why it matters: Rasterization lets an image be treated as the vector input required for geometric views of classification.
• Image-vector dimensionality: State how the length of a rasterized image vector is computed from the matrix dimensions, and relate a 224 × 224 image to a 50,176-dimensional input-space point with pixel values in the range 0 to 255.
  Why it matters: The dimensionality makes concrete why a single image can be viewed geometrically as one point in a very high-dimensional space.

Key terms: pixel, image plane, two-dimensional array, matrix, pixel color value, rasterization, rasterized vector, vector, 50,176-dimensional space, pixel values

3.1.2 Classifiers as decision boundaries

This section presents image classification geometrically, with class examples forming clusters in high-dimensional input space and classifiers separating them with decision boundaries.

• Class clusters in input space: Explain why rasterized images from the same object class tend to form a loose cluster in a high-dimensional input space. Relate the clustering to inherent visual commonalities, such as similar colors or shapes, producing somewhat similar pixel values.
  Why it matters: This geometric view explains why it is possible to distinguish image classes from their input vectors.
• Decision boundary classification: Describe a classifier as a hypersurface, or decision boundary, that separates point clusters for the target classes. Explain that an input is classified by determining which side of this surface its point lies on.
  Why it matters: It gives the geometric meaning of what a classifier must learn from labeled examples.
• Linear and nonlinear separators: Distinguish the simple case in which a hyperplane separates two classes from a harder case requiring a nonlinear curved surface, such as a hypersphere. State that surfaces and planes are called hypersurfaces and hyperplanes in more than three dimensions.
  Why it matters: The complexity of the class arrangement determines whether a straight or curved decision boundary is needed.

Key terms: rasterization, high-dimensional space, class, cluster, pixel values, classifier, hypersurface, decision boundary, point clusters, hyperplane

3.1.3 Modeling in a nutshell

This section explains how classifier modeling uses labeled training examples to choose a model family and iteratively fit its parameters so its decision boundary separates classes.

• Training data and classifier modeling: Explain that training data consists of sampled input examples paired with ground-truth class labels, and that modeling a classifier means using only these examples to approximate a decision boundary that can classify new inputs.
  Why it matters: A classifier must generalize from a limited labeled sample rather than rely on knowing every possible input or the true separating surface.
• Architecture selection and parameter fitting: Describe the two stages of modeling: selecting a parametric function family or architecture before its parameters are known, then training by estimating those parameters from the labeled training data.
  Why it matters: Separating the choice of model family from fitting its parameters clarifies what a model designer decides and what training must learn.
• Training error minimization: Trace one training iteration: evaluate the model output for each training input, compare it with the known target to obtain an error, aggregate errors across the dataset, and adjust the parameters iteratively to reduce that aggregate error.
  Why it matters: This error-reduction loop is the basic purpose of training and motivates the gradient methods developed next.
• Linear and nonlinear boundaries: Distinguish simple cases in which a linear model can represent a planar decision boundary from cases requiring a nonlinear separator, such as a spherical boundary, while recognizing that real separating surfaces often have no familiar geometric form.
  Why it matters: The complexity of the class boundary guides whether a simple linear model is adequate or a nonlinear model is needed.

Key terms: training data, training inputs, ground truth, decision boundary, classifying surface, modeling the classifier, model architecture selection, parametric model function, model training, parameters

3.1.4 Sign of the surface function in binary classification

This subsection explains how the sign of a binary classifier's decision-surface function identifies which side of the boundary an input lies on and therefore predicts its class.

• Sign-based binary classification: Explain how the sign of a decision-surface expression φ(x; b) assigns an input vector to one of two classes in binary classification. State that, after estimating the optimal decision surface, the classifier evaluates φ(x; b) for an input and uses whether the result is positive or negative to predict the class.
  Why it matters: This turns a geometric decision boundary into a concrete rule for making a binary class prediction.
• Half-plane sign interpretation: Describe the two-dimensional example y + 2x + 1 = 0: points on the line make the expression zero, while the two half-planes give opposite signs. Relate this line example to the general idea of surfaces in higher-dimensional input spaces.
  Why it matters: The line example provides the geometric intuition for why a sign can distinguish the two sides of a classifier boundary.

Key terms: binary classifiers, decision boundary, decision surface, φ(x; b), sign, input vector, line, half planes, separator, y + 2x + 1 = 0

3.2 Error, aka loss function

This section defines squared-error loss for individual labeled examples and the total training dataset, which training minimizes by adjusting model parameters.

• Per-example squared error: Describe the loss for one labeled training instance as the squared difference between the model output and that instance’s target output.
  Why it matters: A per-example loss turns a model’s prediction mistake into a number that training can reduce.
• Total training loss: Explain that total loss is the overall squared error across the entire training dataset, not the loss of one particular instance, and that training estimates the parameters w and b that minimize this quantity.
  Why it matters: Training needs one dataset-wide objective so parameter changes can be judged by whether they reduce overall error.

Key terms: loss function, squared error, training data instance, input-target pair, model output, target output, total loss, total error, training data set, L(w,b)

3.3 Minimizing loss functions: Gradient vectors

This section introduces iterative supervised-model training as minimizing total loss by updating weights and bias using loss gradients and a learning rate.

• Iterative loss minimization: Describe model training as starting with random weight and bias values and repeatedly adjusting them to reduce the total loss over the training dataset. Explain why minimizing total error, rather than focusing on individual examples, can produce parameters that work well at inference when the training data is well sampled.
  Why it matters: This is the basic training loop behind fitting a supervised learning model.
• Gradient-based parameter updates: Explain that the gradient of loss with respect to the weights and bias is the multidimensional analog of a derivative and gives the direction of maximum change in loss. Describe gradient descent as repeatedly updating the parameters in the loss-decreasing direction, rather than differentiating with respect to the input.
  Why it matters: Gradients provide the information that tells a training algorithm how to change model parameters to reduce error quickly.
• Learning-rate and stopping choices: Describe the learning rate μ as the parameter that sets the length of each update step, with larger values producing longer steps and smaller values producing shorter ones. State that the simple method uses equal-sized steps, while practical training may stop once accuracy is sufficient rather than waiting for a zero loss gradient.
  Why it matters: Step size and stopping criteria determine whether training progresses efficiently and when a model is usable.

Key terms: training, weights, bias, loss, total error, inferencing, gradient, loss gradient, ∇w,b L(w,b), gradient descent

3.3.1 Gradients: A machine learning-centric introduction

This section derives how small changes to model parameters change loss, culminating in the gradient as the vector tool for locating directions of loss change and stationary optima.

• Parameter-focused loss minimization: Explain why training treats loss as a function of the model parameters w while holding the training inputs and ground-truth outputs implicit. State that training seeks parameter changes that make the loss decrease as much as possible.
  Why it matters: This frames model training as searching parameter space for settings that make predictions closer to the known targets.
• One-dimensional loss changes: Describe the local one-dimensional relation δL = (dL/dw)δw and interpret dL/dw as the slope of the tangent to the loss curve. Explain that a loss-reducing small step δw has the opposite sign from the derivative.
  Why it matters: This one-parameter case provides the intuition for choosing parameter updates that lower loss.
• Multidimensional gradient relation: Describe partial derivatives as rates of loss change with respect to one parameter while all others are fixed, and define the gradient as the vector of all partial derivatives. Trace how the total small loss change is δL = ∇L(w)·δw, so equally long parameter displacements can change loss differently when their directions differ.
  Why it matters: The gradient converts many individual parameter sensitivities into one vector that can guide model optimization.
• Gradient directions and optima: Explain that the gradient direction gives the direction of greatest local increase in loss because the gradient–displacement dot product is largest when the vectors align; consequently, a loss-reducing update goes in the opposite direction. State that at an exact smooth maximum or minimum the gradient is zero, which can be used to solve for an optimum.
  Why it matters: This connects gradient geometry to both iterative loss minimization and algebraic solutions for stationary optima.

Key terms: loss function, parameters, weights, bias, ground truth, training inputs, δL, δw, derivative, dL/dw

3.3.2 Level surface representation and loss minimization

This section visualizes loss functions through level contours and surfaces to show why gradients point in the direction of fastest loss increase and how this guides loss minimization.

• Level contour representation: Describe a level contour as a curve in a two-parameter loss domain on which the loss has a constant value, and generalize it to level surfaces or hypersurfaces in higher-dimensional parameter spaces.
  Why it matters: Level representations let you reason about a high-dimensional loss function from its parameter-space geometry.
• Gradient and level-surface geometry: Explain that motion tangent to a level contour or surface produces zero change in the loss, whereas the gradient is normal to that contour or surface and points in the direction of the fastest increase in loss. State that the gradient magnitude corresponds to the magnitude of this fastest change.
  Why it matters: This geometric relationship explains why a gradient supplies directional information for changing model parameters efficiently.
• Geometric loss minimization: Trace loss minimization as iterative movement through parameter space from an arbitrary starting point toward a minimum, choosing a direction that reduces loss. In the radial example, distinguish the outward gradient direction from the radially inward direction required to reach the origin and minimize the loss.
  Why it matters: Training is parameter adjustment to reduce loss, and the loss geometry shows what direction those adjustments should take.

Key terms: level contour, level surface, level hypersurface, domain, loss function, gradient, tangent, normal, loss minimization, parameter space

3.4 Local approximation for the loss function

This section introduces Taylor series as practical local approximations of a loss function when finite parameter updates make the gradient-based linear change estimate inexact.

• Limits of linear loss change: Explain when the linear expression for a loss change is exact and why it becomes only an approximation for practical finite parameter displacements. State that the approximation worsens as the displacement magnitude increases.
  Why it matters: This explains why a gradient describes loss behavior locally rather than perfectly over arbitrary parameter updates.
• Taylor-series local approximation: Describe how a Taylor series approximates a multidimensional loss function near a chosen point using parameter displacements. Explain that the full infinite series is exact, while practical computation uses a finite truncation—typically no more than the second term because further accuracy gains are too small for current floating-point systems to realize.
  Why it matters: Taylor approximation provides the mathematical bridge from a local gradient estimate to more accurate local descriptions of a loss surface.

Key terms: loss function, parameter space, displacement, gradient vector, linear function, Taylor series, multidimensional function, local neighborhood, displacements, second term

3.4.1 1D Taylor series recap

This subsection recaps how a smooth one-dimensional loss curve can be approximated near a point by a Taylor series and usually truncated to its first-order gradient term.

• One-dimensional Taylor expansion: Explain that a Taylor series describes a smooth one-dimensional function near a chosen point using the function value and its derivatives evaluated at that point. State that the expansion represents changes in the loss L as the parameter changes by a displacement δw.
  Why it matters: Taylor expansion supplies the local mathematical model used to reason about loss functions and their minimization.
• First-order loss approximation: Describe why higher-order Taylor terms can usually be dropped for a sufficiently small displacement δw: they contain progressively higher powers of δw and become negligible. State that retaining only the first-order term gives δL = (δw/1!)(dL/dw), the same local linear relation used for the gradient.
  Why it matters: This explains why gradients provide a practical local approximation for changing and minimizing loss.
• Exponential series example: State that the Taylor expansion of e^x around x = 0 has all derivative values equal to 1 at that point, which yields its familiar power-series expansion. Use this example to illustrate how derivatives at the expansion point determine Taylor-series coefficients.
  Why it matters: The exponential example makes the derivative-based construction of a Taylor series concrete.

Key terms: Taylor series, smooth function, derivatives, displacement δw, loss function L(w), first-order approximation, second-order term, δL, δw, dL/dw

3.4.2 Multidimensional Taylor series and the Hessian matrix

This subsection extends the local Taylor approximation of a loss from one parameter to many by using the gradient and Hessian matrix.

• Multidimensional Taylor approximation: Explain how the multidimensional Taylor series estimates the loss at a nearby parameter point using the loss at the original point, a displacement vector, the gradient, and a quadratic Hessian term.
  Why it matters: This approximation connects the local shape of a loss surface to the parameter changes made during training.
• Hessian matrix properties: Describe the Hessian as the multidimensional second-derivative matrix of the loss, and state why it is symmetric when the relevant mixed partial derivatives agree.
  Why it matters: The Hessian captures local curvature, which is the extra information used by a second-order local approximation.
• First- and second-order locality: Distinguish first-order from second-order local loss approximations: for very small displacements the squared-displacement Hessian term becomes negligible, while for somewhat larger local displacements it can improve the approximation; state that this second-order use is hardly ever done in practice here.
  Why it matters: This explains why gradient-based first-order reasoning is usually sufficient for very small training steps despite the existence of curvature information.

Key terms: multidimensional Taylor series, loss, parameter space, displacement vector, gradient, Hessian matrix, second derivatives, mixed partial derivatives, symmetric matrix, first-order approximation

3.5 PyTorch code for gradient descent, error minimization,and model training

This introduction previews practical PyTorch examples that train models by minimizing error with gradient descent and points readers to the complete code.

• PyTorch training examples: Describe that the upcoming material uses PyTorch examples to train models by minimizing errors through gradient descent.
  Why it matters: This frames the code examples as implementations of the loss-minimization ideas developed earlier in the chapter.
• Complete code location: State that complete code for the section is available through the referenced Manning short URL.
  Why it matters: The code link gives readers a way to inspect or run the full implementations alongside the discussion.

Key terms: PyTorch, gradient descent, error minimization, model training, complete code, http://mng.bz/4Zya

3.5.1 PyTorch code for linear models

This section implements a linear PyTorch model and frames its training as learning weights and a bias that minimize summed squared prediction error.

• Parameterized function estimation: Explain how machine learning models an unknown input-to-output function f with a known parameterized function φ and learns its unknown parameter values from observed input-output pairs.
  Why it matters: This is the basic framing that turns a prediction problem into a trainable model.
• Squared batch loss: Describe how the section defines an individual prediction error as squared difference between the model output and target, then sums such errors over a batch into loss L.
  Why it matters: The loss supplies the concrete quantity that training attempts to reduce.
• Linear prediction and gradient updates: State that the linear model predicts an output as the input vector dotted with the weight vector plus a bias, and explain that parameters begin randomly and are repeatedly adjusted in the direction that reduces error.
  Why it matters: This gives a concrete first model and the training loop that improves it.
• Closed-form gradient limitation: Distinguish the hand-coded closed-form gradients used by the initial linear-model implementation from autograd, which is needed for complicated nonlinear models with many parameters when such gradients are not obtainable in closed form.
  Why it matters: This explains why practical deep-learning frameworks automatically compute gradients rather than requiring manual formulas.

Key terms: true underlying function, input vector, output scalar, output vector, observations, parameterized function, error, squared error, batch error, loss

3.5.2 Autograd: PyTorch automatic gradient computation

This subsection introduces PyTorch Autograd as a scalable way to automatically compute model gradients by tracking computations involving selected variables.

• Autograd scalability rationale: Explain why manually deriving gradients with calculus does not scale to complex models with millions of weights and nonlinear functions, and why an automatic differentiation library such as PyTorch Autograd is used instead.
  Why it matters: Autograd lets practitioners train complex models without having to derive every gradient by hand.
• Gradient tracking computation graph: Describe how to use Autograd by creating a variable with requires_grad=True, and explain that PyTorch records a computation graph as expressions using tracked variables are created and updates it as computation proceeds.
  Why it matters: Gradient tracking and the computation graph are the mechanism that enables PyTorch to calculate gradients from a specified model function.

Key terms: automatic differentiation, PyTorch Autograd, gradient, closed-form gradient, requires_grad = True, tracked variables, computation graph, Autograd

3.5.3 Nonlinear Models in PyTorch

This section contrasts fitting linear and nonlinear models to show that an architecture must match the underlying data relationship, because extra training cannot fix an inadequate linear model.

• Architecture-data relationship match: Explain why a linear model can approximate data generated by a linear underlying output function but fails on a nonlinear data distribution, while a nonlinear model can fit the nonlinear relationship.
  Why it matters: Choosing an architecture that can express the relationship in the data is necessary before training can produce a useful model.
• Architecture limits versus training: Distinguish an inadequate model architecture from insufficient training: state that further training does not make the linear model fit the nonlinear distribution when the model form itself cannot represent the desired function.
  Why it matters: This distinction prevents wasting training effort when the real problem is that the chosen model cannot express the relationship.
• Practical nonlinear architecture choice: State that for real-life problems the usual assumption is that the relationship is nonlinear, so a nonlinear model architecture is chosen accordingly.
  Why it matters: This gives a practical default for selecting models when the true form of a real-world relationship is not already known.

Key terms: linear model, nonlinear model, linear data distribution, nonlinear data distribution, model architecture, underlying output function, inadequate model architecture, further training, nonlinearity, real-life problems

3.5.4 A linear model for the cat brain in PyTorch

This subsection introduces training a PyTorch linear model on the cat-brain dataset and relates its learned parameters to the earlier pseudo-inverse solution.

• PyTorch cat-brain model: Describe that the cat-brain problem is solved here by training a PyTorch model on the same dataset previously used for the direct pseudo-inverse solution.
  Why it matters: This connects a direct linear-algebra solution with training a model through a practical PyTorch implementation.
• Convergent parameter comparison: Explain that, because this is a simple training dataset, the trained model parameters are expected to converge to a solution close to the one obtained by the pseudo-inverse technique.
  Why it matters: It shows that optimization-based training can recover a result similar to a direct solution on a simple problem.

Key terms: cat-brain problem, PyTorch model, pseudo-inverse, training dataset, model parameters, converge, pseudo-inverse technique, simple training dataset

3.6 Convex and nonconvex functions, and global and local minima

This section contrasts convex and nonconvex loss surfaces, explaining why gradient descent reaches a global minimum for convex surfaces but can stop at local minima on nonconvex ones.

• Convex surface optimization: Explain why following the downward gradient on a convex surface reaches its single global minimum regardless of the starting point.
  Why it matters: This explains when gradient-based loss minimization has a guaranteed best destination.
• Local-minimum trapping: Describe how gradient descent can become stuck at a local minimum on a nonconvex surface because the gradient is zero there, even though a better minimum may exist elsewhere.
  Why it matters: Neural-network loss surfaces are nonconvex, so a downhill path is not necessarily a path to the best possible solution.
• Practical local-minimum response: State that neural networks commonly do not use special methods to avoid local minima because a local minimum is often good enough, and explain that retraining from a different random starting point may escape one.
  Why it matters: This gives a practical response when nonconvex training does not reach the best visible solution.

Key terms: convex surface, global minimum, gradient, parameter space, nonconvex surface, local minimum, zero gradient, simulated annealing, random point, retrain

3.7 Convex sets and functions

This introduction frames the study of convex sets and functions, emphasizing tests for convexity, useful properties, and their extension from 2D to higher dimensions.

• Convexity study roadmap: Describe that this section develops ways to determine whether a function is convex and examines useful properties of convex functions. State that the illustrations use two dimensions but the ideas extend to higher-dimensional spaces.
  Why it matters: This roadmap prepares you to connect geometric convexity with optimization and later probability concepts.

Key terms: convex functions, convexity, local minima, 2D space, higher dimensions, Jensen's inequality

3.7.1 Convex sets

This subsection defines convex and nonconvex sets geometrically by whether line segments between their points remain inside the set.

• Convex set definition: Explain that a set is convex when the straight line segment joining any pair of points in the set lies entirely within the set.
  Why it matters: Convexity gives a geometric criterion for recognizing regions with no inward gaps or indentations between their points.
• Nonconvex set criterion: Describe a nonconvex set as one containing at least one pair of its points for which the joining line segment passes through a point outside the set.
  Why it matters: A single segment that exits a region is enough to distinguish a nonconvex set from a convex one.
• Convex boundary property: State that the boundary of a convex set is always a convex curve.
  Why it matters: This links the convexity of a filled region to the geometric character of its enclosing boundary.

Key terms: convex set, straight line, pair of points, nonconvex set, joining line, points C and D, boundary, convex curve

3.7.2 Convex curves and surfaces

This section gives equivalent geometric and differential characterizations of convex curves and surfaces, and distinguishes concavity from nonconvexity.

• Weighted-average convexity criterion: Explain convexity using weighted averages of points on a curve or surface: the weighted average of the function outputs is greater than or equal to the function evaluated at the weighted average of the inputs. Describe the equivalent secant-line picture in which the line segment between two points on a convex curve lies above the curve.
  Why it matters: This criterion generalizes the visual idea of an upward-curving graph to multidimensional functions.
• Curvature and Hessian tests: Describe the differential test for convexity: a one-dimensional function is convex when its curvature, or second derivative, is nonnegative everywhere, and a multidimensional function is convex when its Hessian is positive semidefinite, meaning all Hessian eigenvalues are nonnegative. Distinguish a concave function from a nonconvex one.
  Why it matters: The Hessian provides a practical multidimensional test for whether a smooth loss surface has convex shape.
• Tangent support property: Explain the tangent-line or tangent-plane characterization of convexity: every point of a convex curve or surface lies above a tangent drawn at any point, and the curve or surface touches that tangent at the point of tangency.
  Why it matters: The tangent property provides a local geometric way to recognize the global shape of a convex function.

Key terms: convex function, weighted average, secant line, convex curve, convex surface, second derivative, curvature, Hessian matrix, positive semidefinite, negative semidefinite

3.7.3 Convexity and the Taylor series

This subsection connects Taylor approximations to the two equivalent local characterizations of convexity: positive curvature and lying above tangents.

• Taylor approximation orders: Describe how retaining terms through the first derivative in a one-dimensional Taylor expansion gives a linear, tangent-line approximation, whereas retaining the second-derivative term gives a quadratic approximation.
  Why it matters: Taylor approximations make the local shape of a loss curve concrete enough to reason about its curvature.
• Convexity through Taylor curvature: Explain why an always-positive second derivative makes the quadratic Taylor approximation at least as large as the tangent-line approximation, so locally a convex curve lies above its tangent. Relate this fact to the second-derivative and tangent definitions of convexity.
  Why it matters: This link lets you recognize convexity either from positive curvature or from the geometric behavior of tangents.

Key terms: Taylor expansion, linear approximation, tangent, quadratic approximation, first derivative, second derivative, convex function, tangent definition of convexity, second derivative definition of convexity

3.7.4 Examples of convex functions

This section identifies common convex and concave functions using second derivatives and states operations that preserve convexity.

• Convex function examples: Identify the listed convex functions: g(x)=x², any even power x^(2n) for an integer n, e^x, and −log x; relate their convexity to a positive second derivative where applicable.
  Why it matters: Recognizing standard convex functions helps you determine whether an optimization objective has favorable geometry.
• Logarithm concavity reversal: Distinguish log x, which is concave, from −log x, which is convex, explaining that negating the logarithm reverses the relevant curvature classification.
  Why it matters: This distinction prevents sign mistakes when recognizing convex terms in loss or optimization functions.
• Convexity-preserving operations: State that multiplying a convex function by a positive scalar preserves convexity and that adding convex functions produces another convex function.
  Why it matters: These closure properties let you recognize a larger objective as convex by decomposing it into simple convex pieces.

Key terms: convex, second derivative, even power, positive second derivative, concave, log x, −log x, positive scalar, convexity, sum of convex functions

3.8 Summary

This summary connects geometric classification with loss-based gradient training and the local and global behavior of loss surfaces.