Chapter 2: The mathematical building blocks of neural networks

Introduction

The chapter introduction explains why executable code and interactive notebooks are useful for learning the mathematical ideas behind deep learning, and how to run the book’s examples.

• Executable mathematical descriptions: Explain why the chapter uses executable code rather than relying on mathematical notation to describe operations.
  Why it matters: This explains why the chapter presents mathematical ideas through code.
• Interactive code and notebook setup: Describe how readers can run and experiment with the book’s code, including the recommended hosted option and the local alternative.
  Why it matters: Trying the examples as you read makes the book’s ideas more concrete.

Key terms: mathematical notation, executable code, Jupyter notebook, Google Colaboratory, Colab, GPU runtime

2.1 A first look at a neural network

This section walks through a Keras neural network that learns to classify MNIST handwritten digits, then uses and evaluates it on unseen images.

• MNIST task and data split: Describe the digit-classification task and distinguish the training images and labels from the test images and labels.
  Why it matters: The separate test set lets you check predictions on images the model did not learn from.
• Layers and probability outputs: Explain what layers do in the section’s broad description and what the final layer returns for a digit image.
  Why it matters: This connects the model’s layered structure to the class probabilities used for predictions.
• Compilation choices and roles: Describe the three choices made when compiling the model and the role assigned to each here.
  Why it matters: These choices explain what the model measures, how it improves, and what result is reported.
• Image preparation and fitting: Trace how the images are prepared for the model and how training is started in Keras.
  Why it matters: The input must have the expected shape and scale before the model can be fitted to it.
• Predictions and generalization gap: Explain how the model’s output is used to predict a digit and what its training-versus-test accuracy gap illustrates.
  Why it matters: A high training score alone does not show how well the model classifies new digits.

Key terms: MNIST, training set, test set, samples, labels, classes, layer, representations, Dense layers, fully connected

2.2 Data representations for neural networks

Introduces tensors as the basic data containers of machine learning and relates them to matrices and dimensions.

• Tensors as data containers: Explain that a tensor holds data, usually numbers, and generalizes a matrix to any number of dimensions, also called axes. Describe why tensors matter in machine learning code.
  Why it matters: Manipulating tensors is a routine part of machine learning code.

Key terms: tensor, rank-2 tensor, dimension, axis

2.2.1 Scalars (rank-0 tensors)

Defines a scalar tensor and relates its rank to its number of axes in NumPy.

• Scalar tensor and rank: Explain that a scalar tensor contains one number and has no axes, so its rank and NumPy ndim are both 0.
  Why it matters: This gives you a starting point for describing tensors by their axes.

Key terms: scalar, scalar tensor, rank-0 tensor, 0D tensor, axes, rank

2.2.2 Vectors (rank-1 tensors)

Defines a vector as a one-axis tensor and distinguishes the number of entries on that axis from a tensor’s rank.

• Vector size versus tensor rank: Explain that a vector is an array of numbers with one axis, and distinguish a 5-dimensional vector from a rank-5 tensor.
  Why it matters: Keeping axis count separate from axis size prevents confusion when describing tensor shapes.

Key terms: vector, rank-1 tensor, 1D tensor, axis, rank, dimensionality

2.2.3 Matrices (rank-2 tensors)

Defines a matrix as a rank-2 tensor and identifies its rows and columns as its two axes.

• Matrix axes and entries: Explain that a matrix is an array of vectors, also called a rank-2 or 2D tensor, and identify rows and columns as its first and second axes.
  Why it matters: Rows and columns give you a way to describe how data is organized in a two-dimensional tensor.

Key terms: matrix, rank-2 tensor, 2D tensor, axes, rows, columns

2.2.4 Rank-3 tensors and higher-rank tensors

The section explains how stacking matrices creates rank-3 tensors and how stacking tensors produces higher ranks.

• Constructing higher-rank tensors: Explain that packing matrices into a new array creates a rank-3 tensor, and packing rank-3 tensors creates a rank-4 tensor. Describe a rank-3 tensor as a cube of numbers.
  Why it matters: This gives you a way to picture how tensors extend beyond matrices.

Key terms: rank-3 tensor, 3D tensor, rank-4 tensor

2.2.5 Key attributes

This section defines the three attributes of a tensor and uses MNIST images to show how those attributes describe real data.

• Three key tensor attributes: Explain how rank, shape, and data type describe a tensor, including how rank differs from shape.
  Why it matters: These attributes let you describe what a tensor contains and how its values are arranged.
• MNIST image tensor description: Describe what the rank, shape, and data type of the MNIST training images mean in this example.
  Why it matters: The example connects tensor attributes to the organization of an image dataset.

Key terms: rank, ndim, shape, dtype, MNIST, train_images, rank-3 tensor, 8-bit integers, grayscale image

2.2.6 Manipulating tensors in NumPy

This section shows how to select ranges and image patches from NumPy tensors by slicing their axes.

• Slices across tensor axes: Explain how tensor slicing selects elements along an axis or a range along each axis, including an excluded stop index and the meaning of `:`.
  Why it matters: Slicing lets you select particular images or regions of images without losing track of their axes.
• Negative index positions: Describe how negative indices specify positions relative to the end of the current tensor axis when selecting image patches.
  Why it matters: End-relative positions give you another way to specify image crops.

Key terms: tensor slicing, axis, start index, stop index, negative indices, current axis

2.2.7 The notion of data batches

Explains how deep learning data is divided into batches and why a batch tensor’s first axis is called the batch axis.

• Samples and batch axis: Explain that the first axis of a data tensor represents samples, and that models process fixed-size groups of samples called batches. Describe why the first axis of a batch tensor is called the batch axis.
  Why it matters: It helps you interpret the shape of data passed to a deep learning model.

Key terms: axis 0, samples axis, batch, batch size, batch axis, batch dimension

2.2.8 Real-world examples of data tensors

This section shows how common real-world datasets become tensors, with each axis representing samples, features, time, spatial dimensions, channels, or video frames.

• Vector dataset axes: Explain why a batch of vector data is a rank-2 tensor and identify its samples and features axes.
  Why it matters: Reading the two axes correctly lets you tell how many examples and features a dataset contains.
• Sequence dataset axes: Describe how ordered data is represented as a rank-3 tensor, including the conventional position of the time axis.
  Why it matters: An explicit time axis preserves which observations belong to each ordered sample.
• Image tensors and channel conventions: Explain the axes of batched image tensors, including grayscale channels and the difference between channels-last and channels-first shapes.
  Why it matters: The same images can have different valid tensor shapes depending on where a tool expects the channel axis.
• Video tensor hierarchy: Trace how color image frames, sequences of frames, and batches of videos correspond to tensors of increasing rank.
  Why it matters: Thinking of video as batched sequences of images makes its five axes easier to interpret.

Key terms: vector data, samples axis, features axis, rank-2 tensor, timeseries data, sequence data, timesteps, features, rank-3 tensor, image tensors

2.3 The gears of neural networks: Tensor operations

The section introduces tensor operations as the building blocks of neural-network transformations and illustrates them with a Dense layer.

• Tensor operations as building blocks: Explain that transformations learned by deep neural networks can be expressed as a small set of operations applied to tensors of numeric data.
  Why it matters: This gives you a way to understand a network in terms of the operations its layers perform.
• Dense layer tensor transformation: Trace the Dense-layer example from an input matrix through a tensor product with W, addition of b, and relu to an output matrix that represents the input anew.
  Why it matters: The example connects an actual layer to the tensor operations it performs.

Key terms: tensor operations, tensor functions, tensors, Dense layer, tensor product, matmul, W, b, relu

2.3.1 Element-wise operations

This section defines element-wise tensor operations and explains why vectorized implementations run much faster than naive Python loops.

• Independent tensor entry operations: Explain that an element-wise operation acts independently on each tensor entry, with ReLU and addition as examples.
  Why it matters: This identifies the kind of tensor work that can be performed on many entries in parallel.
• Vectorized execution and speed: Explain why element-wise operations suit parallel execution and contrast naive Python loops with the optimized implementations described for NumPy and GPUs.
  Why it matters: It explains why using optimized array operations can be far faster than writing Python loops.

Key terms: element-wise operations, relu, addition, vectorized implementations, NumPy, BLAS, CUDA

2.3.2 Broadcasting

Explains how broadcasting lets element-wise operations combine tensors of different shapes without physically repeating data.

• Broadcasting shapes and axes: Explain how broadcasting makes a smaller tensor compatible with a larger one, using the section’s (32, 10) matrix and (10,) vector example.
  Why it matters: Broadcasting explains how a Dense layer can add a vector to a rank-2 tensor despite their different shapes.
• Virtual repetition in broadcasting: Distinguish the conceptual repetition used to explain broadcasting from what happens in memory.
  Why it matters: This distinction lets you use the repeated-row picture without assuming broadcasting copies the data.

Key terms: broadcasting, broadcast axes, shape, ndim, element-wise operations, virtual, algorithmic level, memory level

2.3.3 Tensor product

This section explains how tensor products combine vectors and matrices, including the shape requirements and outputs of common matmul operations.

• Vector product and compatibility: Explain that the product of two vectors is a scalar and requires vectors with the same number of elements; identify matmul as the tensor-product operation.
  Why it matters: This is the simplest case of the tensor product used throughout the section.
• Matrix–vector product and order: Describe a matrix–vector product as producing a vector with one coefficient from the product of the vector and each matrix row. Explain why reversing the arguments cannot generally be treated as the same operation.
  Why it matters: It connects the vector product to operations involving matrices and shows why argument order matters.
• Matrix product shape rules: State when two matrices can be multiplied and describe the result’s shape and entries using rows of the first matrix and columns of the second.
  Why it matters: The shape rule lets you check whether a matrix product is valid and predict its output without calculating every entry.

Key terms: tensor product, dot product, matmul, vector, matrix-vector product, rows, ndim, shape compatibility, columns, matrix product

2.3.4 Tensor reshaping

Defines tensor reshaping and identifies matrix transposition as a common special case.

• Tensor reshaping and shape: Explain that reshaping rearranges a tensor’s rows and columns to reach a target shape while preserving its total number of coefficients.
  Why it matters: This explains the shape change used to prepare the digits data for the first model.
• Matrix transposition: Describe transposition as a special case of reshaping that exchanges a matrix’s rows and columns.
  Why it matters: It gives a concrete example of changing a tensor’s shape by rearranging its entries.

Key terms: tensor reshaping, target shape, coefficients, transposition, matrix, rows, columns

2.3.5 Geometric interpretation of tensor operations

This section interprets tensor operations as geometric transformations and explains why Dense networks need activation functions to produce nonlinear transformations.

• Vector addition as translation: Explain how adding a vector moves a point, and how adding that same vector to every point of an object translates the object without distorting it.
  Why it matters: It gives tensor addition a concrete geometric meaning.
• Matrix products as transformations: Describe how matrix products can represent rotation, scaling, and more general linear transforms.
  Why it matters: It connects matrix multiplication to visible changes in geometric space.
• Affine form of Dense layers: Explain why a Dense layer without an activation function is an affine transform, identifying the roles of its matrix product and vector addition.
  Why it matters: It shows what a Dense layer does geometrically before an activation is applied.
• Activations and nonlinear depth: Justify why stacking Dense layers without activations is equivalent to one Dense layer, and why activation functions such as relu are needed for nonlinear transformations.
  Why it matters: It explains why activations, not merely additional Dense layers, enable richer transformations.

Key terms: vector addition, translation, rotation, scaling, linear transform, matrix product, affine transform, Dense layer, y = W @ x + b, activation function

2.3.6 A geometric interpretation of deep learning

This section interprets deep learning as a chain of simple geometric transformations that gradually makes tangled data easier to separate.

• Neural Networks as Geometric Transformations: Explain how a neural network can be viewed as a complex transformation of data in a high-dimensional space, built from a chain of simple tensor operations.
  Why it matters: This gives you a geometric way to picture what a network does to its input.
• Layered Disentangling of Data: Describe how the crumpled-paper analogy illustrates deep learning: successive layers gradually disentangle folded data so that classes can become cleanly separable.
  Why it matters: The paper analogy explains why stacking layers can help separate classes that begin tangled together.

Key terms: tensor operations, geometric transformation, high-dimensional space, data manifolds, uncrumpling, disentangles, classes, layers

2.4 The engine of neural networks: Gradient-based optimization

This section explains how a neural network learns by repeatedly measuring prediction error and using gradients to adjust its weights efficiently.

• Weights as learned information: Explain that a layer’s weights are trainable tensors that hold information learned from training data, and that they begin as small random values before training adjusts them.
  Why it matters: It clarifies what training changes and why the model does not start out making useful predictions.
• Batch training loop: Trace the repeated training steps from selecting a batch and its targets through predictions, loss calculation, and a weight update that reduces that batch’s loss.
  Why it matters: The loop connects a model’s predictions and errors to the changes that constitute learning.
• Cost of coefficient trials: Explain why trying different values for each weight coefficient could indicate a loss-reducing change but is impractical for a model with many coefficients.
  Why it matters: The cost of individual trials motivates the gradient-based alternative.
• Gradient based weight updates: Explain why differentiable model operations allow a gradient to guide changes to the weights and how this permits a loss-reducing update to all coefficients at once.
  Why it matters: This is the section’s explanation of how training can improve many weights without testing them individually.

Key terms: weights, trainable parameters, kernel, bias, random initialization, training loop, batch, targets, forward pass, predictions

2.4.1 What’s a derivative?

This section explains derivatives as local slopes of smooth functions and shows how their direction guides minimization.

• Local linear approximation: Explain why a smooth function can be approximated by a line near a point, so a sufficiently small change in x produces an approximately proportional change in y.
  Why it matters: The local line is what gives the derivative its meaning.
• Derivative as local slope: Describe what the derivative at a point says about the direction and speed of change, and what the derivative function f′(x) returns.
  Why it matters: The derivative tells you how a function responds to a small change at a particular input.
• Derivative-guided minimization: Explain which direction to move x by a small amount when using a derivative to reduce f(x).
  Why it matters: This connects derivatives to the optimization used in neural-network training.

Key terms: smooth, local linear approximation, slope, derivative, derivative function, magnitude of the derivative, optimization, minimize, opposite direction from the derivative

2.4.2 Derivative of a tensor operation: The gradient

This section extends derivatives to tensor-valued inputs and shows how the gradient of a loss with respect to model weights suggests an update that reduces the loss.

• Gradients of tensor functions: Explain that a gradient generalizes a derivative to a function with tensor inputs and describes how its output changes as those inputs vary.
  Why it matters: This connects the familiar idea of a derivative to the tensor operations used in neural networks.
• Weight gradient components: Describe the gradient of loss with respect to a weight matrix W at W0, including its shape and the meaning of each coefficient.
  Why it matters: It makes the gradient concrete as information about individual model weights.
• Small opposite-gradient updates: Explain how moving a weight tensor a small amount opposite its loss gradient can reduce loss, and why the step must be small.
  Why it matters: This is the section’s link between measuring how loss changes and adjusting weights to lower it.

Key terms: gradient, tensor function, derivative, W, W0, loss_value, partial derivative, gradient of loss_value with respect to W, step

2.4.3 Stochastic gradient descent

This section explains why neural networks use gradient-based updates instead of solving for optimal weights analytically, then describes mini-batch SGD, its practical choices, and momentum.

• Limits of analytical minimization: Explain why solving directly for the weights that minimize a neural network’s loss is impractical, even though the loss is differentiable.
  Why it matters: It explains why training proceeds through incremental updates rather than a direct solution.
• Mini-batch SGD update cycle: Trace one mini-batch SGD update from a randomly drawn batch through predictions, loss, gradient, and a small parameter change opposite the gradient.
  Why it matters: This is the update cycle that turns a loss measurement into changes to a network’s weights.
• Learning rate tradeoff: Explain how the learning rate scales a gradient update and what can happen when it is too small or too large.
  Why it matters: The gradient gives a direction, but the learning rate determines whether steps along it are useful.
• Gradient descent batch sizes: Distinguish true SGD, mini-batch SGD, and batch gradient descent by how much data each uses per update, and explain the stated efficiency tradeoff.
  Why it matters: These names describe a practical choice about the cost and accuracy of each update.
• Loss surface visualization limits: Describe what one- and two-dimensional loss illustrations represent and why they may give misleading intuitions about real network training.
  Why it matters: It keeps the useful downhill picture from being mistaken for a literal view of network training.
• Momentum and local minima: Explain the local-minimum problem illustrated here and how momentum uses a previous update as well as the current gradient to address it.
  Why it matters: Momentum shows why an optimizer might consider the history of its movement rather than only the slope at its current position.

Key terms: loss function, gradient, parameters, mini-batch stochastic gradient descent, forward pass, backward pass, loss, learning rate, local minimum, true SGD

2.4.4 Chaining derivatives: The Backpropagation algorithm

This section explains how backpropagation applies the chain rule to a computation graph to calculate a neural network’s loss gradients, and how frameworks automate that process.

• Chain rule for gradients: Explain how derivatives of simple tensor operations can be combined with the chain rule to find the gradient of a network’s loss with respect to its weights.
  Why it matters: This explains how a complex network can have computable gradients even though it contains many operations.
• Forward and backward graph passes: Trace how the section’s scalar computation graph first produces a loss in a forward pass, then uses a backward pass and the chain rule to calculate gradients for w and b.
  Why it matters: The small graph shows concretely what it means to propagate a loss contribution backward to each parameter.
• Computation graphs and automatic differentiation: Explain why representing operations as a computation graph lets modern frameworks provide gradients without hand-writing backpropagation.
  Why it matters: It connects the hand-worked graph to why frameworks can calculate gradients for you.

Key terms: chain rule, backpropagation, gradient, atomic operations, computation graph, forward pass, backward pass, loss_val, w, b

2.5 Looking back at our first example

The section revisits the first handwritten-digit example to connect its data, model, loss, optimizer, and training call to the neural-network concepts explained earlier.

• Input tensors and model weights: Describe how the first example represents its images and where its model stores what it learns.
  Why it matters: It connects the example’s data and model code to the tensors and weights the network actually uses.
• Prediction loss and optimizer roles: Explain how the model, loss function, and optimizer relate in the first example, including the roles of sparse_categorical_crossentropy and adam.
  Why it matters: These roles explain how predictions provide feedback for changing the model rather than simply producing an answer.
• Fit batches and gradient updates: Trace what the fit call does with batches and epochs, and how each batch leads to a weight update.
  Why it matters: It makes the fit call understandable as repeated, batch-by-batch changes to the model’s weights.

Key terms: NumPy tensors, float32, Dense layers, weight tensors, loss function, sparse_categorical_crossentropy, optimizer, adam, mini-batch stochastic gradient descent, fit

2.5.1 Reimplementing our first example from scratch

The section sketches a low-level reimplementation of the first neural-network model using a simple Dense layer and a class that chains layers.

• Dense Layer Components: Describe how the simple Dense layer is represented: it has Keras variables W and b as model parameters, and its __call__() method applies an input transformation with an element-wise activation.
  Why it matters: It connects the mathematical description of a layer to the objects and method used in a concrete implementation.
• Sequential Layer Chaining: Explain how NaiveSequential forms a model by holding a list of layers, calling them on the input in order, and exposing a weights property to track their parameters.
  Why it matters: It shows how individual layers become one callable model whose parameters can be tracked.

Key terms: Dense layer, W, b, model parameters, activation, NaiveDense, NaiveSequential, layers, __call__(), weights

2.5.2 Running one training step

This section explains how one batch of data produces a loss, gradients, and a small weight update, using TensorFlow’s automatic differentiation to obtain the gradients.

• Per-batch training step: Trace the four operations in one training step: predict from a batch, compute loss against its labels, find the loss gradient with respect to the weights, and update the weights opposite to that gradient.
  Why it matters: This sequence is the unit of work repeated during training.
• Learning-rate weight update: Explain that the simplest weight update subtracts gradient times learning rate from each weight, and that in practice a Keras Optimizer usually handles the update.
  Why it matters: It connects the gradient to the actual change in model weights.
• GradientTape automatic differentiation: Describe how TensorFlow’s tf.GradientTape records tensor operations in a computation graph that can be used to obtain a scalar value’s gradient with respect to input values, avoiding a hand-written gradient function in this example.
  Why it matters: It explains how the example obtains loss gradients without implementing backpropagation by hand.

Key terms: training step, batch, loss, gradient, weights, weight update, learning rate, Optimizer, automatic differentiation, tf.GradientTape

2.5.3 The full training loop

Defines an epoch and the full training loop in terms of repeated training steps.

• Epoch and training loop structure: Explain that an epoch repeats the training step for each batch in the training data, and the full training loop repeats epochs.
  Why it matters: This gives you the basic structure for describing how model training proceeds.

Key terms: training step, batch, epoch, full training loop

2.5.4 Evaluating the model

The section evaluates the handwritten model on test images and explains why working through its low-level steps helps readers understand Keras.

• Test prediction comparison: Describe how to evaluate the model by taking the argmax of its predictions for test images and comparing the resulting labels with the expected labels.
  Why it matters: This connects the model’s predictions to a check against the test labels.
• Handwritten steps and Keras: Explain why working through the low-level steps is useful despite taking more work than a few lines of Keras code.
  Why it matters: The handwritten work makes Keras’s shorter workflow less of a black box.

Key terms: argmax, predictions, test images, expected labels, by hand, fit(), low-level mental model, Keras API

2.6 Summary