Chapter 2: Vectors, matrices, and tensors in machine learning

Introduction

This chapter introduction frames machine learning as organized numerical computation and previews a machine-learning-focused study of vectors, matrices, eigenvalues, eigenvectors, and hyperellipses with accompanying executable notebooks.

• Organized numerical representations: Explain why machine learning requires numbers to be organized into meaningful objects rather than processed randomly, and identify vectors and matrices as such objects for inputs and outputs.
  Why it matters: Vectors and matrices provide the basic language used to represent and transform machine-learning data.
• Chapter mathematical roadmap: Describe the chapter's stated progression from basic vectors and matrices to machine-learning-relevant advanced topics, including eigenvalues, eigenvectors, and finding the axes of a hyperellipse.
  Why it matters: The roadmap helps you place each later linear-algebra topic in the chapter's overall machine-learning purpose.
• Companion notebook availability: State that most concepts in this chapter and other chapters have Jupyter Notebook-based Python implementations, with complete downloadable code intended to run after Python and Jupyter Notebook are installed.
  Why it matters: The companion code gives you a way to execute and inspect implementations of the mathematical concepts.

Key terms: vectors, matrices, number crunching, eigenvalues, eigenvectors, hyperellipse, Jupyter Notebook, Python, complete, fully functional code

2.1 Vectors and their role in machine learning

This section introduces vectors as ordered numerical representations of machine-learning inputs, outputs, parameters, and object features.

• Vectors as ordered representations: Explain that a vector is an ordered sequence, or array, of two or more numbers used to compactly represent an entity. Describe how the position of each number gives it a particular meaning, such as hardness first and sharpness second for the cat-brain input.
  Why it matters: Machine-learning models handle many related measurements by packaging them into vectors with a defined ordering.
• Vectors throughout machine learning: Describe the machine-learning quantities represented as vectors: model inputs, model outputs, the inputs and outputs of neural-network layers, and typically model parameters. Explain the object-recognition output example as a three-entry vector whose entries represent probabilities for dog, human, and cat in a fixed order.
  Why it matters: Recognizing vectors in each part of a model makes it easier to follow how neural networks pass and transform information.
• Feature-vector construction: Explain that a feature vector quantitatively describes properties of an object for a particular machine-learning problem. Describe the document example: choose interesting words, exclude ubiquitous words such as “and,” “if,” and “to,” and use each selected word's occurrence count as a vector entry; also recognize an image's pixel sequence as a feature vector.
  Why it matters: Feature vectors turn real-world objects such as documents and images into numerical forms that models can compare and process.

Key terms: vector, array, ordered sequence, input vector, output vector, model parameters, neural network layer, feature vector, document retrieval, natural language processing

2.1.1 The geometric view of vectors and its significance in machine learning

This section presents vectors as geometric points or positions, connecting coordinate representations to the geometric view of machine-learning models.

• Vectors as geometric positions: Explain how a vector can represent the position of a point relative to an origin, while an array of coordinate values quantitatively represents that vector in a chosen coordinate system. Describe the arrowed-line mental picture illustrated by the position from O to P.
  Why it matters: This distinction lets you reason about data geometrically rather than treating vectors as only lists of numbers.
• Coordinate-system dependence: Distinguish the abstract vector, which represents the position of P relative to O independently of coordinate axes, from its array representation, which depends on the chosen axes and reference point. State that machine-learning computations must use one fixed coordinate system consistently or appropriately transform vectors when coordinates change.
  Why it matters: Coordinate consistency prevents numerical vector representations from being combined as though they described the same geometric space when they do not.
• Geometric model mapping: Describe how n-element vectors represent points in n-dimensional spaces and how this lets a machine-learning model be viewed as a geometric transformation mapping input points to output points in high-dimensional spaces. Explain why relative positions, rather than absolute positions, matter for the loss-minimization focus of machine learning.
  Why it matters: Viewing a model as a transformation of points supplies a recurring geometric language for understanding machine-learning methods.
• High-dimensional geometric intuition: State that machine learning often uses spaces with thousands of dimensions, which cannot usually be visualized directly, and describe the qualified use of two- or three-dimensional analogies: they are often useful but can fail in higher dimensions.
  Why it matters: Useful geometric analogies can aid reasoning, but recognizing their limits avoids overextending low-dimensional intuition.

Key terms: vector, coordinate values, coordinate system, origin, array representation, coordinate axes, reference point, transform, n-dimensional space, input points

2.2 PyTorch code for vector manipulations

This section introduces PyTorch as the book’s practical deep-learning tool and explains how readers should engage with its Jupyter Notebook code.

• PyTorch's role in the book: Describe PyTorch as the open-source machine learning library used for the book’s practical deep-learning code, alongside the mathematical material.
  Why it matters: PyTorch is the programming environment through which the book connects mathematical ideas to working deep-learning applications.
• Notebook-based code study: State that the book’s Python code is produced in Jupyter Notebook and that a theoretical summary precedes each code snippet.
  Why it matters: Knowing how code examples are presented helps you connect each implementation to the idea it demonstrates.
• Reader preparation expectations: Describe the expected preparation for using the examples: basic Python knowledge is assumed, appropriate packages such as PyTorch must be installed, and readers are strongly encouraged to try every snippet.
  Why it matters: Running the examples yourself is the intended way to turn the mathematical discussion into practical familiarity.

Key terms: PyTorch, open source machine learning library, deep learning applications, Python, Jupyter Notebook, code snippet, theoretical material, Python basics, packages, code snippets

2.2.1 PyTorch code for the introduction to vectors

Demonstrates PyTorch operations for creating vectors and accessing, selecting, and slicing their elements.

• PyTorch vector manipulation: Describe that Listing 2.1 demonstrates creating a vector in PyTorch and accessing individual elements, subvectors, and slices of a vector.
  Why it matters: These operations let you inspect and select parts of vector data when implementing machine-learning computations.

Key terms: vectors, subvectors, slice and dice, PyTorch, Listing 2.1

2.3 Matrices and their role in machine learning

This section introduces matrices as rectangular arrays that compactly represent a dataset of many feature-vector input instances.

• Matrix structure and vector views: Describe a matrix as a rectangular array of numbers with a fixed number of rows and columns, and explain that either every row or every column can be viewed as a vector.
  Why it matters: Matrices provide the standard compact way to organize many related numerical values for machine learning.
• Training dataset matrix representation: Explain how a collection of training inputs is represented by a matrix X: each row is one input instance, while each column corresponds to one feature element across the instances.
  Why it matters: This row-per-example and column-per-feature convention makes it possible to express machine learning data and computations compactly.

Key terms: matrix, rectangular array, rows, columns, row vectors, column vectors, training dataset, training input, input instance, matrix X

2.3.1 Matrix representation of digital images

This subsection explains how a grayscale digital image is stored as a matrix whose entries encode pixel brightness.

• Grayscale image matrix representation: Explain how a digital grayscale image can be represented as a matrix, with each matrix element corresponding to the brightness of the pixel at a particular x,y position.
  Why it matters: This representation lets machine-learning methods treat image pixels as numerical data.
• Brightness value encoding: Describe the stated grayscale brightness encoding: values are typically normalized to integers from 0 through 255, where 0 is black, 255 is white, and 128 is gray.
  Why it matters: Knowing what the numbers mean makes an image matrix interpretable as visible light and dark pixels.
• Example image indexing and gradient: Trace the example 9-pixel-wide by 4-pixel-high image: brightness increases from left to right and from top to bottom, with I00 as the black top-left pixel and I3,8 as the white bottom-right pixel.
  Why it matters: The example shows how matrix indices locate pixels and how their values form a visible image pattern.

Key terms: matrix, digital image, pixel, brightness, normalized, 0 to 255, black, white, gray, I00

2.4 Python code: Introducing matrices, tensors, and images via PyTorch

This section presents tensors in PyTorch as multidimensional arrays and maps scalars, vectors, matrices, RGB images, and image batches to their tensor dimensions and shapes.

• Tensor dimensionality hierarchy: Describe tensors as multidimensional arrays for programming, and state how scalars, vectors, and matrices correspond to zero-, one-, and two-dimensional tensors.
  Why it matters: This gives you the basic language for recognizing what kind of numerical object a PyTorch program is manipulating.
• Image tensor shapes: Explain that an RGB image is a three-dimensional tensor organized as color channels × height × width, and that a batch of 64 such images is a four-dimensional tensor with shape 64 × color channels × height × width.
  Why it matters: Image shape conventions determine how image data must be arranged for PyTorch-based deep learning operations.

Key terms: tensor, multidimensional array, scalar, vector, matrix, RGB image, color channels, height, width, batch

2.5 Basic vector and matrix operations in machine learning

This section introduces basic vector and matrix operations through application-oriented examples in image processing, computer vision, and machine learning rather than as a comprehensive linear algebra review.

• Application-centric linear algebra scope: Describe the purpose and scope of this section as an application-focused introduction to basic vector and matrix operations for image processing, computer vision, and machine learning, rather than a complete linear algebra treatment.
  Why it matters: This frames the operations as practical tools for representing and manipulating machine learning data.

Key terms: vector operations, matrix operations, image processing, computer vision, machine learning, linear algebra

2.5.1 Matrix and vector transpose

This section defines matrix transposition as swapping rows and columns, including the row-vector form of a transposed column vector.

• Matrix and vector transposition: Describe matrix transposition as exchanging a matrix’s row and column indices, including the resulting change from an m × n matrix to an n × m matrix. Explain that transposing a column vector produces its corresponding row vector because a vector is a one-column matrix.
  Why it matters: Transpose notation lets you reorient data and express later vector and matrix operations correctly.

Key terms: matrix transposition, transpose, Aᵀ, row indices, column indices, column vector

2.5.2 Dot product of two vectors and its role in machine learning

This section defines the dot product and shows how it compactly expresses the weighted-sum output of a simple machine-learning model.

• Dot product definition: Define the dot product of two vectors as the sum of the products of their corresponding elements, and state that the two vectors must have the same number of elements.
  Why it matters: The dot product is the basic operation that combines an input's feature values with a model's weights.
• Weighted-sum model notation: Explain that a simple model computes its output by taking the dot product of an input feature vector and a weight vector, then adding a bias: y = x · w + b.
  Why it matters: This notation is the compact mathematical form of a linear model used throughout machine learning.
• Compact representation rationale: Justify using dot-product notation by explaining that explicitly writing every feature-weight product becomes unwieldy for long feature vectors, whereas x · w + b stays compact regardless of the number of inputs.
  Why it matters: Compact vector notation makes large machine-learning models readable and manipulable.

Key terms: dot product, corresponding elements, same dimensions, inner product, input vector, weight vector, bias, model output, weighted sum, y = x · w + b

2.5.3 Matrix multiplication and machine learning

This section shows how matrix multiplication compactly computes weighted model outputs across datasets, generalizes to matrix products, and relates products to dot products and transposes.

• Batched weighted-sum outputs: Explain how multiplying a data matrix X by a weight column vector computes one weighted-sum output for every input instance at once, with each output equal to the dot product of a row of X and the weight vector. State that adding the bias produces the model output vector compactly for the whole training dataset.
  Why it matters: Matrix-vector multiplication lets a model process many training examples in one compact operation.
• Matrix product construction rules: Describe when two matrices can be multiplied and how to determine the result shape: an m-by-p matrix times a p-by-n matrix yields an m-by-n matrix. Explain that each result entry is the dot product of a row of the left matrix with a column of the right matrix, and state that matrix multiplication is generally not commutative.
  Why it matters: These shape and entry rules are essential for reading and constructing the matrix computations used by machine-learning models.
• Product special cases and transpose: Explain that a vector dot product is a special case of matrix multiplication obtained by transposing one vector, and that row-vector–matrix multiplication is another special case. State the transpose rule for a valid product: the transpose of AB equals the product of the individual transposes in reversed order, and that this also applies to matrix-vector products.
  Why it matters: Recognizing these equivalences makes matrix expressions easier to transform and manipulate correctly.

Key terms: matrix-vector multiplication, weight vector, bias, weighted sum, training dataset, matrix-matrix multiplication, m by p matrix, p by n matrix, row vector, column vector

2.5.4 Length of a vector (L2 norm): Model error

This section defines model error as squared output-target differences and shows that total training error is the squared L2 norm of the residual vector.

• Squared error rationale: Explain why a model’s output error is commonly squared: outputs equally far above and below the target should receive the same nonnegative error, and model comparison usually needs relative rather than exact error magnitudes.
  Why it matters: Squared error turns signed prediction differences into a consistent quantity for judging how well model parameters perform.
• Single-instance model error: State how squared error is computed for one training instance: form the model output, subtract the ground-truth target, and square the difference; for a biased weighted-sum model, the output is the weight vector dotted with the input plus a bias.
  Why it matters: This is the per-example quantity that reveals whether one choice of weights and bias predicts closer to the target than another.
• Training residual L2 norm: Describe how squared error extends from one example to an entire training dataset: collect model outputs and targets into vectors, form their difference (the residual vector), and sum the squares of its elements. Explain that this total squared training error equals the residual vector dotted with itself, its squared magnitude or L2 norm.
  Why it matters: The L2 norm gives one total error value for evaluating model parameters across all training examples.

Key terms: squared error, target value, computed value, model output, ground truth, biased weighted sum model, weight vector, bias, target, output vector

2.5.5 Geometric intuitions for vector length

This subsection gives a geometric interpretation of vector length, introduces unit vectors as normalized directions, and relates error-vector length to machine-learning training.

• Geometric L2 norm: Explain the L2 norm of a 2D vector as the hypotenuse of a right triangle whose side lengths are the vector’s elements, and state that this length intuition extends to higher dimensions.
  Why it matters: This geometric view makes vector length easier to interpret when measuring quantities such as model error.
• Unit-vector normalization: Describe a unit vector as a vector of length 1 and explain that it is obtained from a nonzero vector by dividing every element by that vector’s length. State that unit vectors typically represent direction.
  Why it matters: Normalization separates a vector’s direction from its magnitude, which is useful when only direction should matter.
• Error-vector minimization: State that an error vector is the difference between a model-output vector and the target ground-truth vector, and explain that training often aims to minimize the length of this error vector.
  Why it matters: It connects a geometric quantity—vector length—to the practical objective of making model predictions match targets.

Key terms: L2 norm, vector length, unit vector, length, direction, hat symbol, error vector, model output vector, target ground truth vector, training

2.5.6 Geometric intuitions for the dot product: Feature similarity

This section explains how the dot product measures directional agreement and similarity between feature vectors, including why normalization removes vector-length effects.

• Dot product feature similarity: Explain how a dot product can provide an initial similarity score for document feature vectors. Describe why documents with many common words of interest have a larger dot product, whereas documents with no common such words can have a dot product of zero.
  Why it matters: Dot products provide a basic way to compare feature-based representations such as documents.
• Vector component and projection: Describe the dot product geometrically as the component of one vector in the direction of another. State that this relation is expressed as the product of the vectors' lengths and the cosine of the angle between them.
  Why it matters: The projection interpretation explains why the dot product captures how much two vector directions agree.
• Directional agreement and normalization: Explain how the angle between vectors determines the sign and directional-agreement part of their dot product: same direction gives maximum positive agreement, perpendicular vectors give zero, and opposite directions give a negative value. Distinguish the ordinary dot product, which also increases with vector lengths, from the normalized dot product of unit vectors, which is neutral to length.
  Why it matters: Separating directional similarity from vector magnitude is essential when comparing feature vectors fairly.
• Unit-vector difference relation: Relate the dot product of two unit vectors to the distance between them: explain that a larger dot product corresponds to a smaller difference vector and therefore greater directional agreement.
  Why it matters: This gives a distance-based intuition for why aligned unit vectors are similar.

Key terms: feature vector, dot product, similarity, document retrieval, component, direction, cosine, angle, vector length, directional agreement

2.6 Orthogonality of vectors and its physical significance

This section uses the dot product to connect vector angle with physical effectiveness and defines orthogonal vectors as perpendicular, independent vectors with no shared component.

• Dot product and directional effectiveness: Explain how the dot product ||u||||v||cosθ relates the angle between a force-like vector and a displacement-like vector to how effective the force is. State that aligned vectors produce the maximum effect, increasing angle reduces the effect, and a 90-degree angle produces zero effect.
  Why it matters: This gives a physical and geometric intuition for why the dot product measures alignment between vectors.
• Orthogonal vector independence: Describe orthogonal vectors as vectors whose dot product is zero and which are geometrically perpendicular. Explain the section's physical interpretation that orthogonal vectors are independent and cannot influence one another, illustrated by document feature vectors with no words in common.
  Why it matters: Orthogonality provides a way to recognize directions or features that have no overlap or mutual effect.

Key terms: dot product, angle, cosθ, force vector, displacement vector, orthogonal, perpendicular, feature vector

2.7 Python code: Basic vector and matrix operations via PyTorch

This section introduces a PyTorch notebook that illustrates the vector and matrix operations discussed earlier.

• PyTorch operations notebook: Describe the purpose of this section's PyTorch code: illustrating previously discussed vector and matrix concepts, with fully functional code available as an executable Jupyter Notebook.
  Why it matters: The notebook provides a practical way to inspect and run the linear-algebra operations used in machine learning.

Key terms: PyTorch, Jupyter Notebook, vector and matrix operations

2.7.1 PyTorch code for a matrix transpose

2.7.2 PyTorch code for a dot product

This subsection shows the PyTorch dot-product operation for two vectors and restates its component-wise mathematical definition.

• Dot product computation: State that the dot product of vectors [a1, a2, a3] and [b1, b2, b3] is the scalar a1b1 + a2b2 + a3b3, and describe it as representing the components of one vector along the other.
  Why it matters: Dot products quantify how vectors relate and are a basic operation used throughout machine learning.

Key terms: dot product, vectors, components

2.7.3 PyTorch code for matrix vector multiplication

This subsection demonstrates that multiplying an m-by-n matrix by an n-element vector produces an m-element column vector in PyTorch.

• Matrix-vector output dimensions: State the dimensionality rule for matrix-vector multiplication: an m-by-n matrix multiplied by an n-element vector produces an m-element column vector, as in the example with m=3 and n=2.
  Why it matters: Tracking dimensions lets you determine whether a matrix-vector operation is valid and what shape its result will have.

Key terms: matrix-vector multiplication, m-by-n matrix, m-element column vector

2.7.4 PyTorch code for matrix-matrix multiplication

This subsection states the dimension rule and output shape for multiplying two matrices in PyTorch.

• Matrix multiplication dimensions: Describe the required dimensions for multiplying an m-by-p matrix A by a p-by-n matrix B, and state that the result C is an m-by-n matrix.
  Why it matters: Checking the shared inner dimension lets you predict whether a matrix multiplication is valid and what shape it produces.

Key terms: matrix-matrix multiplication, m-by-p matrix, p-by-n matrix, m-by-n matrix

2.7.5 PyTorch code for the transpose of a matrix product

This subsection states and demonstrates the transpose rule for a valid matrix product in PyTorch.

• Transpose of matrix products: State that when matrices A and B can be multiplied, the transpose of their product is the product of their transposes in reverse order: (AB)^T = B^T A^T. Explain that reversing the order is part of the rule.
  Why it matters: This identity lets you rearrange transposed matrix expressions correctly in later linear-algebra and neural-network calculations.

Key terms: transpose, matrix product, reversed order, (AB)^T = B^T A^T

2.8 Multidimensional line and plane equations and machine learning

This section frames machine learning classification geometrically as separating feature-vector points with high-dimensional lines, planes, or more complex surfaces, and introduces the upcoming study of hyperlines and hyperplanes.

• Geometric view of classification: Explain how classifier inputs are feature vectors viewed as points in a multidimensional feature space, and how classification requires separating points from different classes.
  Why it matters: This geometric viewpoint connects abstract classifier behavior to the arrangement of data in feature space.
• Feature-space transformation: Describe the model’s role in transforming feature-vector points from an input space, where class points may be jumbled, into an output space where the classes are easier to separate.
  Why it matters: Models can make classification possible by changing the representation of data rather than relying on an already-separable input arrangement.
• High-dimensional separators: Distinguish the geometrical forms of decision separators described here: a line in a simple two-dimensional case, a line or plane in high-dimensional cases, and a curved surface in more complicated cases; state that high-dimensional lines and planes are called hyperlines and hyperplanes.
  Why it matters: Recognizing the possible shapes of separators prepares you to understand the mathematical treatment of hyperlines and hyperplanes.

Key terms: classifier, feature vectors, multidimensional feature space, classes, input space, output space, transform, separator, line, plane

2.8.1 Multidimensional line equation

This section expresses a line between two vectors with a parameterized vector equation that works in any finite-dimensional space.

• Multidimensional line equation: Describe the parameterized equation for the line joining two vectors: start at one vector and travel in the direction given by the difference between the second and first vectors, using an arbitrary scalar α.
  Why it matters: This vector form replaces the two-dimensional slope-intercept equation with one that works in any number of dimensions.
• Alpha ranges and line segments: Explain how the value of α locates a point relative to the two endpoint vectors: values from 0 to 1 give points between them, negative values extend past the first vector, and values greater than 1 extend past the second vector.
  Why it matters: The parameter range distinguishes interpolation between endpoints from extensions of the full line.

Key terms: multidimensional line equation, direction, α, weighted combination, line segment

2.8.2 Multidimensional planes and their role in machine learning

This section explains hyperplanes as multidimensional decision boundaries and connects their geometry to the weights and bias of a simple machine learning classifier.

• Hyperplanes as class separators: Explain how a plane separates buy and no-buy stock examples in three-dimensional feature space, and how this generalizes to a hyperplane when there are more input variables. Describe why a hyperplane may fail to separate intertwined real-world data without first transforming the inputs into another space.
  Why it matters: A classifier can be understood geometrically as a boundary that divides different decision regions.
• Normal-vector plane equation: Describe a hyperplane using a fixed point on it and a normal direction: displacement vectors from the fixed point to every other point on the plane are orthogonal to the normal. State that this condition leads to a dot-product equation for a plane or hyperplane.
  Why it matters: The normal-vector view explains both the orientation of a decision boundary and its algebraic equation.
• Weights, bias, and sides: Explain that the hyperplane equation wᵀx + b = 0 is the weighted-sum-plus-bias model viewed geometrically: the weight vector sets the hyperplane's normal orientation and the bias sets its location. State how the sign of wᵀx + b identifies which side of the hyperplane an input lies on, while zero means the input is on the boundary.
  Why it matters: This is the geometric interpretation of what training learns when it adjusts weights and biases in a linear classifier.

Key terms: plane, hyperplane, feature space, class separator, transformation, input space, normal direction, normal vector, orthogonal, fixed point

2.9 Linear combinations, vector spans, basis vectors, and collinearity preservation

This introduction motivates studying how high-dimensional spaces can be represented using a small set of vectors.

• Compact vector representation: Explain the central questions of whether every point in a high-dimensional space can be expressed using a few vectors and what the smallest such set would be.
  Why it matters: These questions lead to the ideas used to describe the structure and dimensionality of vector spaces.

Key terms: high-dimensional spaces, vectors

2.9.1 Linear dependence

This subsection defines linear dependence through nonzero scalar-weighted sums that equal zero, connecting it to collinearity and linear combinations.

• Linear dependence criterion: State that vectors are linearly dependent when scalar weights, not all zero, can be chosen so that their weighted sum equals the zero vector. Illustrate the criterion with the example that [1 1] and [2 2] are dependent because -2[1 1] + [2 2] = 0.
  Why it matters: Linear dependence identifies when some vectors fail to contribute an independent direction or piece of information.
• Collinearity and dependence: Describe the geometric claim that collinear vectors are linearly dependent. Trace the proof idea: a point on the line can be expressed using a scalar parameter from the multidimensional line equation, then rearranging that equation gives a nonzero linear combination equal to zero.
  Why it matters: The geometric view makes the algebraic zero-sum definition easier to recognize in a picture.
• Linear combination definition: Define a linear combination as a weighted sum of vectors, such as α1v1 + α2v2 + … + αnvn, where the weights are scalars. Explain that linear dependence is established by finding a particular linear combination equal to zero with weights not all zero.
  Why it matters: Linear combinations are the algebraic language used to express dependence and later to describe spans and bases.

Key terms: linear dependence, zero vector, scalar weights, collinearity, collinear vectors, multidimensional line equation, linear combination, weighted sum

2.9.2 Span of a set of vectors

This subsection defines the span of a vector set as all of its linear combinations and illustrates that two Cartesian axis vectors span a 2D plane.

• Vector-set span: Explain that the span of a set of vectors is the set of every vector obtainable as a linear combination of the original vectors, including the original vectors themselves.
  Why it matters: Span specifies the region of vector space that a collection of vectors can represent.
• Cartesian axes spanning a plane: Describe how two vectors along the Cartesian X- and Y-axes span their entire 2D plane because any vector in that plane can be written as a weighted sum of those two axis vectors.
  Why it matters: This example makes span concrete by showing how component weights locate any point or vector in a plane.

Key terms: span, linear combination, set of vectors, Cartesian coordinate axes, X-axis, Y-axis, weighted sum

2.9.3 Vector spaces, basis vectors, and closure

This section defines vector spaces, bases, dimensional limits on independent vectors, and closure under linear combinations through algebraic and geometric examples.

• Vector space definition: Explain that a vector space is a set of vectors in which vector addition and scalar multiplication are defined, so linear combinations of its members can be formed.
  Why it matters: Vector spaces provide the mathematical setting in which machine-learning data vectors and their combinations are analyzed.
• Basis vectors and coordinates: Describe a basis as a set of vectors that spans a vector space, so every vector in the space can be written as a linear combination of the basis vectors. Explain that the standard basis of ℝn corresponds to Cartesian coordinate axes, but bases need not be orthogonal or unique.
  Why it matters: A basis supplies the directions used to represent every vector in a space, including the familiar coordinate-axis representation.
• Dimensional basis limit: State that an n-dimensional space needs exactly n basis vectors, and that any collection of more than n vectors in ℝn is linearly dependent. Relate this to n being the largest possible number of linearly independent vectors in an n-dimensional space.
  Why it matters: This dimensional limit tells you when a set of vectors contains redundant directions rather than independent ones.
• Closure under combinations: Explain closure under linear combination: every linear combination of vectors from a set must remain in that same set. Describe why ℝ2, viewed as a plane, is closed, whereas points only on a sphere's surface are not closed because connecting arbitrary surface points does not stay on the surface.
  Why it matters: Closure distinguishes spaces that can safely contain the results of their own vector combinations from sets that cannot.

Key terms: vector space, vector addition, scalar multiplication, linear combination, basis, basis vectors, span, Cartesian coordinate axes, orthogonal, linearly independent

2.10 Linear transforms: Geometric and algebraic interpretations

This section introduces matrix-based transforms as ways to map feature vectors into more useful spaces and identifies collinearity preservation as the defining property of linear transforms.

• Transform motivation in feature spaces: Explain why machine learning systems transform feature-vector points into other spaces. Describe how a transform can make classes more segregated for classification or simplify data by removing axes with little variation.
  Why it matters: Transforming a representation can make the structure needed for a machine learning task easier to analyze.
• Matrix transform mappings: Describe how an m × n matrix can map an n × 1 input vector to an m-dimensional output vector through matrix multiplication. Give the section's examples: R maps ℝ2 to ℝ2 by rotating a point 45 degrees, while P maps ℝ3 to ℝ2 by projecting a 3D point onto the X-Y plane.
  Why it matters: Matrix transforms provide the algebraic language for changing data representations, including transformations that preserve or change dimensionality.
• Collinearity-preserving linear transforms: Explain that a linear transform preserves collinearity: points on a straight line before transformation remain on a possibly different straight line afterward. State that this class of transforms can always be represented by matrix multiplication, and conversely every matrix multiplication is a linear transformation.
  Why it matters: Collinearity preservation is the geometric property that connects linear transformations to matrix multiplication.

Key terms: feature vector, feature space, classifier, transform, matrix multiplication, R : ℝ2 → ℝ2, P : ℝ3 → ℝ2, rotation, projection, linear transform

2.10.1 Generic multidimensional definition of linear transforms

This section defines a linear transform as a function that preserves linear combinations of vectors.

• Linear transform definition: Explain that a transform φ is linear exactly when applying it to a linear combination of two vectors gives the same result as taking the same weighted linear combination of their individual transforms. Describe rotation- and projection-matrix multiplication as examples of linear transforms.
  Why it matters: This preservation rule is the defining property that lets linear transformations be analyzed with linear algebra.

Key terms: linear transform, linear combination, rotation matrix, projection matrix

2.10.2 All matrix-vector multiplications are linear transforms

This subsection proves that finite-dimensional matrix-vector multiplication is a linear transform and contrasts it with the nonlinear vector-length function.

• Matrix multiplication linearity: Explain why multiplying vectors by a matrix satisfies the linear-map rule: applying a matrix to a weighted sum of vectors gives the same result as taking the corresponding weighted sum of their matrix products.
  Why it matters: This identifies matrix-vector multiplication as the algebraic form of a linear transformation.
• Finite-dimensional matrix equivalence: State the finite-dimensional relationship between matrices and linear transforms: every matrix multiplication is a linear transform, and every linear transform on finite-dimensional vectors can be represented by a matrix; note that this equivalence does not extend to transforms on infinite-dimensional vectors.
  Why it matters: It lets you interpret a matrix both as an array of numbers and as a geometric operation on vectors.
• Vector length nonlinearity: Describe why the function that maps a vector to its length is nonlinear: even with both scalar weights equal to one, the length of a vector sum need not equal the sum of the individual lengths, so it violates the linear-map rule.
  Why it matters: This provides a concrete test showing that not every function on vectors is a linear transformation.

Key terms: linear transform, linear mapping, matrix-vector multiplication, finite dimensions, matrix, infinite-dimensional vectors, geometric view, vector length, length of a vector, nonlinear mapping

2.11 Multidimensional arrays, multilinear transforms, and tensors

Introduces tensors as a concept commonly encountered in machine learning.

• Tensor concept introduction: State that this section introduces the concept of a tensor, a term commonly encountered in machine learning.
  Why it matters: Tensors are a recurring representation concept in machine learning and its software tools.

Key terms: tensor, machine learning, TensorFlow

2.11.1 Array view: Multidimensional arrays of numbers

This subsection introduces tensors as multidimensional numerical arrays, connecting scalars, vectors, matrices, color images, and neural-network layer data.

• Tensor array interpretation: Describe the provisional array view of a tensor as a generalized n-dimensional array, while noting that multidimensional arrays and tensors are not strictly identical. State that a scalar, vector, and matrix can be viewed respectively as 0-, 1-, and 2-tensors.
  Why it matters: This provides the vocabulary for describing the numerical data that deep-learning models manipulate.
• Tensor examples in deep learning: Explain why a color RGB image is a multidimensional array or tensor: it combines separate red, green, and blue images. State that the inputs and outputs of every neural-network layer are also tensors.
  Why it matters: Images and layer activations must be represented as tensors for neural networks to process them.

Key terms: tensor, multidimensional array, scalar, vector, matrix, color image, RGB, R, G, B

2.12 Linear systems and matrix inverse

This section introduces linear systems as compact matrix equations for estimating model parameters and relates their closed-form solution to matrix inverses.

• Iterative and closed-form estimation: Distinguish the usual iterative estimation of machine-learning parameters from closed-form solution methods for smaller problems. Explain why large training datasets and parameter sets generally make one-step closed-form solutions impractical, while traditional methods can be faster and more accurate for small problems.
  Why it matters: This distinction explains why direct linear-algebraic solutions are useful even though most machine learning training is iterative.
• Matrix form of linear systems: Describe how the weighted-sum cat-brain constraints can be represented as a linear system, with a matrix of input values and bias terms multiplying a vector of unknown weights and bias to produce a target-output vector. State that the compact form A x = b represents a linear system in any number of unknowns.
  Why it matters: Writing many equations as A x = b makes high-dimensional machine-learning parameter problems compact and manageable.
• Matrix inverse and identity: Explain the inverse-based closed-form solution of a square linear system as x = A⁻¹b, by analogy with the scalar solution of ax = b. State that an inverse exists when det(A) is nonzero and that multiplying a matrix by its inverse yields the identity matrix I, which leaves matrices and vectors unchanged.
  Why it matters: The inverse and identity provide the basic algebraic intuition for directly solving suitable systems of linear equations.

Key terms: iterative process, closed-form solution, training data, machine parameters, linear system, matrix multiplication, weighted sum model, weights, bias, A x = b

2.12.1 Linear systems with zero or near-zero determinants,and ill-conditioned systems

This section explains why zero determinants make square linear systems singular and unsolvable, while near-zero determinants make them ill-conditioned and numerically unstable.

• Singularity equivalences: Explain the equivalent signs that a square-matrix linear system is singular: dependent rows or columns, a zero determinant, a nonexistent inverse, and inability to solve the system exactly.
  Why it matters: These equivalent tests reveal when matrix inversion cannot provide a valid solution to a linear system.
• Redundant-equation example: Describe how one equation that is a scalar multiple of another creates linearly dependent matrix rows and therefore fails to supply two independent equations.
  Why it matters: Recognizing duplicated constraints makes the abstract idea of singularity visible directly in a system of equations.
• Ill-conditioned systems: Explain why a determinant close to zero makes a theoretically solvable system ill-conditioned: inverse entries can become large, so a small change in the input vector can cause a drastic change in the solution.
  Why it matters: Ill-conditioning warns that an algebraically solvable problem may still yield unreliable numerical results.

Key terms: singular matrix, zero determinant, linearly dependent rows or columns, matrix inverse, linear system, scalar multiple, linearly dependent, row vectors, near singularity, near-zero determinant

2.12.2 PyTorch code for inverse, determinant, and singularity testing of matrices

This subsection shows that PyTorch's linalg package computes matrix inverses and determinants, and uses a zero determinant to identify a singular, non-invertible matrix.

• PyTorch inverse and determinant calls: Describe that PyTorch's linalg package can compute a matrix inverse and its determinant with function calls.
  Why it matters: These operations let you check and manipulate matrices when solving linear-algebra problems in code.
• Singularity determinant test: Explain that a matrix is singular when its determinant is zero, which means it has no inverse and a linear system using it cannot be solved.
  Why it matters: Testing for singularity prevents attempting an inverse-based solution where no matrix inverse exists.

Key terms: PyTorch, linalg, matrix inverse, determinant, singular matrix, zero determinant, non-invertible, linear system of equations

2.12.3 Over- and under-determined linear systems in machine learning

This section relates nonsquare linear systems to machine learning training data and explains why noisy overdetermined systems require minimizing overall error.

• Nonsquare systems in machine learning: Explain how a machine learning linear system can have a nonsquare matrix A because training-data instances determine the number of equations while model weights determine the number of unknowns. State that these quantities are independent and that such systems are often solved iteratively.
  Why it matters: Machine learning normally has a number of data constraints that does not exactly match the number of model parameters.
• Overdetermined and underdetermined cases: Distinguish the two nonsquare cases for an m × n matrix: an overdetermined system has m > n, or more equations than unknowns, whereas an underdetermined system has m < n, or fewer equations than unknowns.
  Why it matters: Comparing equations with unknowns identifies the kind of nonsquare problem a model-fitting task presents.
• Optimal fitting of noisy equations: Describe why an overdetermined machine learning system can be inconsistent: noisy training data can prevent any one set of unknowns from satisfying every equation. Explain that the objective is therefore to find values that minimize the overall error, making A times the unknown vector as close as possible to the target vector.
  Why it matters: This shift from satisfying every equation exactly to minimizing aggregate error is the basis of fitting models to real noisy data.

Key terms: nonsquare matrix, training data instances, equations, unknowns, weights, iteratively, m × n, overdetermined system, underdetermined system, m > n

2.12.4 Moore Penrose pseudo-inverse of a matrix

This subsection derives the Moore–Penrose pseudo-inverse as a way to solve nonsquare linear systems by converting them through a transpose operation.

• Pseudo-inverse derivation: Derive the pseudo-inverse expression A+ = (AᵀA)⁻¹Aᵀ for a possibly nonsquare matrix A by multiplying a linear system by Aᵀ and then inverting the resulting square matrix AᵀA, assuming it is invertible.
  Why it matters: The pseudo-inverse extends the familiar inverse-based solution idea to linear systems whose coefficient matrix is not square.
• Nonsquare system solution: Explain that the pseudo-inverse provides a solution x = A+b for overdetermined or underdetermined linear systems when the usual inverse A⁻¹ is generally unavailable, and state that this solution minimizes the error norm ||Ax − b||.
  Why it matters: Many machine learning problems lead to systems with too many or too few equations, where an exact ordinary-inverse solution is not available.
• Ordinary inverse limitation: State why the usual inverse A⁻¹ cannot in general be used for a not-necessarily-square m × n matrix: a nonsquare matrix does not in general have the determinant or ordinary inverse required by the standard inverse solution.
  Why it matters: Recognizing when ordinary inversion is unavailable explains why a different construction is needed.

Key terms: pseudo-inverse, Moore Penrose pseudo-inverse, A+, transpose, AᵀA, overdetermined system, underdetermined system, error, ||Ax − b||, determinant

2.12.5 Pseudo-inverse of a matrix: A beautiful geometric intuition

This section derives the Moore-Penrose pseudo-inverse geometrically as a projection-based least-squares solution and applies it to fitting an overdetermined linear model.

• Column-space least-squares view: Explain a linear system A x = b as seeking a vector A x in the column space of A that minimizes the distance ||A x - b||. Distinguish the invertible square case, where b can be matched exactly, from a nonsquare case, where the closest attainable column-space point is sought.
  Why it matters: This view turns an unsolvable exact system into a well-defined best-fit problem.
• Projection normal equation: Trace the geometric derivation in which b is projected onto the column space of A, making the residual b - A x perpendicular to that space. State that orthogonality to every vector A z in the column space yields A^T(b - A x) = 0, the normal equation.
  Why it matters: The perpendicular-residual condition explains why the pseudo-inverse produces the closest available output.
• Augmented cat-brain fitting: Describe how the overdetermined cat-brain training system is written as X w = y after augmenting every input row with a trailing 1 to include the bias. State that, because there are 15 training examples but only w0, w1, and b to determine, the pseudo-inverse w = (X^T X)^-1 X^T y gives the best-effort fit that minimizes total training error.
  Why it matters: Augmentation and least squares let a linear model fit all its weights and bias in one matrix calculation.

Key terms: column space, linear span, least-squares solution, best fit, total error, projection, residual, orthogonal, normal equation, Moore-Penrose pseudo-inverse

2.12.6 PyTorch code to solve overdetermined systems

2.13 Eigenvalues and eigenvectors: Swiss Army knives of machine learning

This section introduces eigenvalues and eigenvectors as direction-preserving properties of square-matrix transforms, motivating their use for dimensionality reduction and interpreting them through 3D rotation.

• Eigenvectors in dimensionality reduction: Explain why eigenvalues and eigenvectors help simplify high-dimensional feature data: they identify axes of variation so axes with scant variation can be eliminated when mapping data to a friendlier lower-dimensional space.
  Why it matters: These tools later provide a way to find broad patterns in large, high-dimensional datasets.
• Eigenvector direction preservation: Describe an eigenvector of a square matrix A as a nonzero vector whose direction does not change under the matrix's linear transformation, expressed by A v = λv; explain that the eigenvalue λ may change its length while remaining parallel to the original vector.
  Why it matters: The direction-preserving interpretation makes eigenvectors a concrete physical property of a linear transformation rather than just an algebraic calculation.
• Characteristic equation and rotation example: Trace how nonzero eigenvectors require A − λI to be non-invertible, giving det(A − λI) = 0, and state that an n × n matrix consequently has n eigenvalues counting solutions, though they need not be distinct. Describe the rotation example: the rotation axis stays fixed, is the eigenvector for eigenvalue 1, and in 3D the other two eigenvalues encode the rotation angle and can be complex.
  Why it matters: This connects the geometric idea of a preserved direction to the determinant equation used to calculate eigenvalues.

Key terms: feature vectors, feature space, dimensionality reduction, eigenvalues, eigenvectors, square matrix, linear transformation, eigenvector, eigenvalue, A v = λv

2.13.1 Eigenvectors and linear independence

This subsection proves that eigenvectors associated with distinct eigenvalues of the same matrix must be linearly independent.

• Distinct-eigenvalue independence: Explain why two eigenvectors of a matrix with unequal eigenvalues are linearly independent. Trace the contradiction proof that begins by assuming a nontrivial linear dependence between the two eigenvectors.
  Why it matters: This result lets distinct eigenvalue directions serve as genuinely separate directions in later matrix analysis.

Key terms: eigenvector, eigenvalue, linearly independent, linearly dependent, matrix A

2.13.2 Symmetric matrices and orthogonal eigenvectors

This subsection proves that eigenvectors of a symmetric matrix associated with distinct eigenvalues are orthogonal.

• Orthogonality of symmetric eigenvectors: State that eigenvectors of a symmetric matrix belonging to different eigenvalues are mutually orthogonal. Express this as v_i^T v_j=0 when λ_i≠λ_j.
  Why it matters: This property lets symmetric matrices be analyzed using perpendicular eigenvector directions.
• Transpose-based orthogonality proof: Trace the proof that symmetry produces two scalar expressions for the same eigenvector dot product, λ1v1^Tv2 and λ2v1^Tv2. Explain that subtracting them gives (λ1-λ2)v1^Tv2=0, so distinct eigenvalues force v1^Tv2=0.
  Why it matters: Following this argument connects the algebraic condition A^T=A directly to the geometric result of perpendicular eigenvectors.

Key terms: symmetric matrix, eigenvectors, eigenvalues, mutually orthogonal, distinct eigenvalues, matrix symmetry, A^T=A, transpose, eigenvalue-eigenvector pair, dot product

2.13.3 PyTorch code to compute eigenvectors and eigenvalues

2.14 Orthogonal (rotation) matrices and their eigenvalues and eigenvectors

This section introduces rotation, or orthogonal, matrices as intuitively observable transforms that also support machine learning tools.

• Rotation matrices overview: Describe rotation transforms as orthogonal matrices and state why the section highlights them: their behavior is readily observable in the mechanical world and they are important for developing and analyzing machine learning tools.
  Why it matters: Rotation matrices provide an intuitive kind of transformation that will be examined through their orthogonality and eigenstructure.

Key terms: rotation transforms, rotation matrices, orthogonal matrices

2.14.1 Rotation matrices

This section introduces standard 2D and 3D rotation matrices and relates each 3D rotation’s unchanged coordinate axis to an eigenvector with eigenvalue 1.

• Planar rotation matrix: Describe planar rotation about the origin by angle θ using the standard 2D rotation matrix, and explain that rotation preserves a point’s distance from the origin while changing its coordinates from (x, y) to (x′, y′).
  Why it matters: Rotation matrices provide a concrete example of a linear transformation with a clear geometric meaning.
• Three-dimensional axial rotations: State how the standard 3D rotation matrices rotate about the Z-, X-, and Y-axes: each applies the 2D sine-cosine rotation to the plane perpendicular to its axis while leaving that axis’s coordinate unchanged.
  Why it matters: Axis-specific rotations are the basic transformations used to represent orientation and motion in three dimensions.
• Rotation-axis eigenvectors: Explain why the coordinate axis of a 3D rotation is an eigenvector with eigenvalue 1: a point on that axis maps to itself because its axis coordinate is unaffected by the rotation.
  Why it matters: This connection lets eigenvectors reveal the fixed axis of a geometric transformation.

Key terms: planar rotation, rotation matrix, origin, angle θ, rotation about the Z-axis, rotation about the X-axis, rotation about the Y-axis, Z coordinate, X coordinate, Y coordinate

2.14.2 Orthogonality of rotation matrices

This section defines orthogonal rotation matrices, shows that they preserve vector length, and explains why their transpose reverses the rotation angle.

• Orthogonal rotation matrices: State the defining identity for an orthogonal matrix, namely that its transpose is its inverse, so both RᵀR and RRᵀ equal the identity matrix. Describe the section's relationship between orthogonal matrices and rotation matrices.
  Why it matters: This identity makes rotation matrices easy to recognize and manipulate without separately computing an inverse.
• Length preservation under rotation: Explain why applying an orthogonal rotation matrix R to a vector preserves its length: the squared length of Rv reduces to vᵀRᵀRv, and RᵀR = I leaves vᵀv unchanged.
  Why it matters: Length preservation distinguishes a pure rotation from transformations that stretch or shrink vectors.
• Transpose as reverse rotation: Describe how to reverse a rotation by angle θ: rotate by −θ, using the inverse rotation matrix. Explain that for an orthogonal rotation matrix this inverse is Rᵀ, so transposing the matrix reverses the rotation.
  Why it matters: This lets you undo a rotation directly from its matrix, a useful operation in geometric transformations.

Key terms: orthogonal matrix, rotation matrix, transpose, inverse, identity matrix, length-preserving, magnitude, orthogonality, L2 norm, negative angle

2.14.3 PyTorch code for orthogonality of rotation matrices

This subsection verifies in PyTorch that a rotation matrix is orthogonal by multiplying it by its transpose and checking for the identity matrix.

• PyTorch orthogonality verification: Describe how to verify a rotation matrix's orthogonality in PyTorch: create the rotation matrix, transpose it, and multiply the original matrix by its transpose. State that the result should be the identity matrix.
  Why it matters: This gives a direct computational check that a rotation matrix has the defining orthogonality property.

Key terms: orthogonality, rotation matrix, transpose, identity matrix

2.14.4 Eigenvalues and eigenvectors of a rotation matrix:Finding the axis of rotation

This subsection shows that a rotation matrix has eigenvalue 1 and that its associated eigenvector identifies the rotation axis.

• Rotation eigenvalue derivation: Trace how the eigenvalue equation for a rotation matrix, together with transposition and multiplication by the corresponding quantities, leads to eigenvalue solutions of 1 or −1. State that the −1 solution corresponds to reflection.
  Why it matters: This connects the algebraic eigenvalue equation to the geometric distinction between rotation and reflection.
• Rotation axis eigenvector: Explain that 1 is an eigenvalue of a rotation matrix and that its corresponding eigenvector is unchanged by the rotation. Describe this unchanged direction or set of fixed points as the axis of rotation.
  Why it matters: Eigenvectors turn a rotation into a geometric object you can identify directly: its fixed axis.

Key terms: rotation matrix, eigenvalue, eigenvector, reflection, eigenvalue 1, axis of rotation

2.14.5 PyTorch code for eigenvalues and vectors of rotation matrices

2.15 Matrix diagonalization

This section introduces matrix diagonalization through a matrix’s linearly independent eigenvectors and shows why it supports solving systems without inversion and later machine-learning tools.

• Eigenvector matrix diagonalization: Describe how an n × n matrix A with n linearly independent eigenvectors can be expressed using a matrix S whose columns are those eigenvectors and a diagonal matrix Λ containing the corresponding eigenvalues.
  Why it matters: Diagonalization turns a general matrix problem into a simpler diagonal-matrix problem.
• Symmetric matrix diagonalization: Explain why a symmetric matrix can be diagonalized as A = SΛSᵀ: its eigenvectors are orthogonal, so the eigenvector matrix satisfies SᵀS = SSᵀ = I and S⁻¹ = Sᵀ.
  Why it matters: For symmetric matrices, orthogonal eigenvectors replace a matrix inverse with a transpose, making the diagonal form especially convenient.
• Diagonalization purpose and nonuniqueness: State that diagonalization offers a way to address linear systems without directly inverting the original matrix, provides insight for quadratic forms and PCA, and is not unique for a given matrix.
  Why it matters: The value of diagonalization is that it simplifies later computations and reveals useful geometric directions in data.

Key terms: diagonalization, linearly independent eigenvectors, eigenvector matrix, diagonal matrix, eigenvalues, S, symmetric matrix, orthogonal eigenvectors, orthogonal matrix, transpose

2.15.1 PyTorch code for matrix diagonalization

This subsection points readers to a PyTorch implementation of the matrix diagonalization mathematics introduced in section 2.15.

• PyTorch matrix diagonalization implementation: Describe that this subsection presents the directly relevant PyTorch code for implementing the matrix diagonalization mathematics developed in section 2.15.
  Why it matters: The code connects the mathematical diagonalization discussion to an executable PyTorch implementation.
• Executable notebook availability: State that fully functional, Jupyter Notebook–executable code for this section is available through the linked online resource.
  Why it matters: The linked notebook lets readers run the complete code rather than seeing only the directly relevant excerpt.

Key terms: PyTorch, matrix diagonalization, Jupyter Notebook, fully functional code

2.15.2 Solving linear systems without inversion via diagonalization

This section shows how diagonalizing a symmetric matrix solves a linear system through stable matrix-vector operations instead of directly inverting the matrix.

• Inversion avoidance rationale: Explain why directly solving a linear system by computing A⁻¹ should be avoided when possible, and why diagonalization offers an alternative for a square symmetric matrix with distinct eigenvalues.
  Why it matters: Avoiding a full matrix inverse makes linear-system computation simpler and more numerically stable.
• Diagonalized system solution: Describe the diagonalization-based steps for solving Ax=b: write A as SΛSᵀ, multiply the right-hand side by Sᵀ, solve the resulting diagonal system by inverting Λ entry by entry, and multiply by S to recover x.
  Why it matters: This procedure replaces one difficult inverse with orthogonal coordinate changes and an easy diagonal solve.

Key terms: matrix inversion, numerically unstable, diagonalization, square symmetric matrix, distinct eigenvalues, linear system, S, Λ, eigenvectors, orthogonal

2.15.3 PyTorch code for solving linear systems via diagonalization

This subsection sets up a linear system for solution via diagonalization by identifying the orthogonal eigenvector structure of its symmetric coefficient matrix.

• Symmetric matrix eigenvector structure: Explain why the coefficient matrix A in the example has orthogonal eigenvectors: it is symmetric. State that placing those eigenvectors in the columns of a matrix produces an orthogonal matrix.
  Why it matters: Orthogonal eigenvectors make diagonalization-based linear-system computations especially simple.
• Transpose-inverse equivalence: State that for the orthogonal matrix formed from A's eigenvectors, the transpose equals the inverse. Relate this property to the diagonalization-based setup for solving the displayed system.
  Why it matters: Replacing an inverse with a transpose is the key simplification provided by an orthogonal eigenvector matrix.

Key terms: symmetric matrix, orthogonal eigenvectors, orthogonal matrix, eigenvectors, transpose, inverse, diagonalization

2.15.4 Matrix powers using diagonalization

This subsection shows that diagonalizing a matrix makes repeated matrix-power computations efficient.

• Diagonalized matrix powers: Explain how to compute the nth power of a diagonalizable matrix A by using its diagonalization A = S D S⁻¹, so that Aⁿ = S Dⁿ S⁻¹. State that the power is applied to the diagonal matrix D rather than by repeatedly multiplying A by itself.
  Why it matters: Diagonalization converts a difficult repeated matrix multiplication into simple operations on diagonal entries.
• Efficiency for repeated powers: Describe why precomputing the diagonalization matrix S is useful when powers of the same m × m matrix are needed at different times. State that the subsection contrasts computing powers with O(m) operations after precomputation against naive computation requiring O(nm³) operations.
  Why it matters: The benefit of diagonalization is especially large when the same matrix must be raised to many powers.

Key terms: diagonalization, diagonalizable matrix, diagonal matrix, matrix power, S, D, precompute, O(m) operations, O(nm³) operations, naive computations

2.16 Spectral decomposition of a symmetric matrix

This section states the spectral resolution theorem: a square symmetric matrix can be decomposed in terms of its eigenvalues and eigenvectors.

• Spectral decomposition of symmetric matrices: Explain that a square symmetric matrix can be written in terms of its eigenvalues and eigenvectors, a representation called its spectral decomposition or spectral resolution. Describe the decomposition as an expression that combines the matrix’s eigenvalues with its corresponding eigenvectors.
  Why it matters: This representation exposes the fundamental directions and scalings encoded by a symmetric matrix.

Key terms: spectral decomposition, spectral resolution theorem, symmetric matrix, eigenvalues, eigenvectors

2.16.1 PyTorch code for the spectral decomposition of a matrix

2.17 An application relevant to machine learning: Finding the axes of a hyperellipse

This section expresses a hyperellipse as a quadratic matrix equation and shows that diagonalizing its symmetric matrix reveals the directions of its axes.

• Hyperellipse quadratic representation: Describe how an origin-centered, axis-aligned ellipse can be written using a position vector and a diagonal matrix, and how this representation extends to an n-dimensional hyperellipse. State how a center offset can be included in the generalized equation.
  Why it matters: Matrix form makes the same geometric object workable in any number of dimensions.
• Rotation and symmetric ellipse matrix: Explain how rotating an axis-aligned hyperellipse produces a generally non-diagonal matrix A in its quadratic equation. State that this matrix is symmetric because it has the form RᵀΛR, with Λ diagonal and R a rotation.
  Why it matters: This explains why a tilted multidimensional ellipse can still be analyzed with symmetric-matrix tools.
• Eigenvectors as hyperellipse axes: Trace how to find the directions of a generic hyperellipse's axes by diagonalizing its symmetric matrix A. Explain that placing A's eigenvectors in the columns of S gives an orthogonal rotation, and in the rotated coordinates the hyperellipse is axis aligned, so those eigenvectors are its axes.
  Why it matters: Eigenvectors convert a tilted hyperellipse into an axis-aligned one and expose its meaningful geometric directions.

Key terms: ellipse, hyperellipse, position vector, diagonal matrix, axis aligned, centered at the origin, rotation, rotation matrix, RᵀΛR, symmetric matrix

2.17.1 PyTorch code for hyperellipses

This subsection shows that diagonalizing the symmetric matrix in a hyperellipse equation rotates the coordinate system so its axes align with the ellipse's principal axes.

• Hyperellipse matrix formulation: Describe the example hyperellipse as a quadratic equation in two coordinates and explain that its middle matrix encodes the cross-term and orientation of the ellipse.
  Why it matters: Writing the ellipse with a matrix makes its orientation and axes accessible through linear algebra.
• Diagonalization for axis alignment: Explain that diagonalizing the matrix A as SΣS⁻¹ removes the off-diagonal coupling in the hyperellipse equation, producing a form with a diagonal matrix Σ in transformed coordinates.
  Why it matters: Diagonal form reveals the independent coordinate directions that define the hyperellipse's axes.
• Orthogonal eigenvector rotation: Trace why symmetry of A makes its eigenvectors orthogonal, so the eigenvector matrix S is orthogonal and S⁻¹ = Sᵀ. Explain that the transformed coordinates are obtained by Sᵀ and that S acts as a rotation aligning coordinate axes with the ellipse axes.
  Why it matters: The orthogonal-eigenvector property explains why finding principal axes is a rotation rather than an arbitrary distortion of the coordinate system.

Key terms: hyperellipse, ellipse equation, matrix, vector, cross-term, diagonalization, A = SΣS⁻¹, diagonal matrix, transformed coordinates, hyperellipse axes

2.18 Summary

This summary consolidates the vector, matrix, linear-system, and eigenvalue tools used to represent data and analyze machine-learning transformations.