Cover of Math and Architectures of Deep Learning

What you will learn in Math and Architectures of Deep Learning

by Krishnendu Chaudhury

Read it on liveBook Get the book at manning.com

Chapter by chapter

Chapter 1: An overview of machine learning and deep learning

This introduction frames deep learning as a transformative practical technology and explains why the book emphasizes mathematical principles over memorizing recipes.

1.1 A first look at machine/deep learning: A paradigm shift in computation · 1.2 A function approximation view of machine learning:Models and their training · 1.3 A simple machine learning model: The cat brain · 1.3.1 Input features · 1.3.2 Output decisions · 1.3.3 Model estimation · 1.3.4 Model architecture selection · 1.3.5 Model training · 1.3.6 Inferencing · 1.4 Geometrical view of machine learning · 1.5 Regression vs. classification in machine learning · 1.6 Linear vs. nonlinear models · 1.7 Higher expressive power through multiple nonlinear layers: Deep neural networks · 1.8 Summary

What you learn in chapter 1 →

Chapter 2: Vectors, matrices, and tensors in machine learning

This chapter introduction frames machine learning as organized numerical computation and previews a machine-learning-focused study of vectors, matrices, eigenvalues, eigenvectors, and hyperellipses with accompanying executable notebooks.

2.1 Vectors and their role in machine learning · 2.1.1 The geometric view of vectors and its significance in machine learning · 2.2 PyTorch code for vector manipulations · 2.2.1 PyTorch code for the introduction to vectors · 2.3 Matrices and their role in machine learning · 2.3.1 Matrix representation of digital images · 2.4 Python code: Introducing matrices, tensors, and images via PyTorch · 2.5 Basic vector and matrix operations in machine learning · 2.5.1 Matrix and vector transpose · 2.5.2 Dot product of two vectors and its role in machine learning · 2.5.3 Matrix multiplication and machine learning · 2.5.4 Length of a vector (L2 norm): Model error · 2.5.5 Geometric intuitions for vector length · 2.5.6 Geometric intuitions for the dot product: Feature similarity · 2.6 Orthogonality of vectors and its physical significance · 2.7 Python code: Basic vector and matrix operations via PyTorch · 2.7.1 PyTorch code for a matrix transpose · 2.7.2 PyTorch code for a dot product · 2.7.3 PyTorch code for matrix vector multiplication · 2.7.4 PyTorch code for matrix-matrix multiplication · 2.7.5 PyTorch code for the transpose of a matrix product · 2.8 Multidimensional line and plane equations and machine learning · 2.8.1 Multidimensional line equation · 2.8.2 Multidimensional planes and their role in machine learning · 2.9 Linear combinations, vector spans, basis vectors, and collinearity preservation · 2.9.1 Linear dependence · 2.9.2 Span of a set of vectors · 2.9.3 Vector spaces, basis vectors, and closure · 2.10 Linear transforms: Geometric and algebraic interpretations · 2.10.1 Generic multidimensional definition of linear transforms · 2.10.2 All matrix-vector multiplications are linear transforms · 2.11 Multidimensional arrays, multilinear transforms, and tensors · 2.11.1 Array view: Multidimensional arrays of numbers · 2.12 Linear systems and matrix inverse · 2.12.1 Linear systems with zero or near-zero determinants,and ill-conditioned systems · 2.12.2 PyTorch code for inverse, determinant, and singularity testing of matrices · 2.12.3 Over- and under-determined linear systems in machine learning · 2.12.4 Moore Penrose pseudo-inverse of a matrix · 2.12.5 Pseudo-inverse of a matrix: A beautiful geometric intuition · 2.12.6 PyTorch code to solve overdetermined systems · 2.13 Eigenvalues and eigenvectors: Swiss Army knives of machine learning · 2.13.1 Eigenvectors and linear independence · 2.13.2 Symmetric matrices and orthogonal eigenvectors · 2.13.3 PyTorch code to compute eigenvectors and eigenvalues · 2.14 Orthogonal (rotation) matrices and their eigenvalues and eigenvectors · 2.14.1 Rotation matrices · 2.14.2 Orthogonality of rotation matrices · 2.14.3 PyTorch code for orthogonality of rotation matrices · 2.14.4 Eigenvalues and eigenvectors of a rotation matrix:Finding the axis of rotation · 2.14.5 PyTorch code for eigenvalues and vectors of rotation matrices · 2.15 Matrix diagonalization · 2.15.1 PyTorch code for matrix diagonalization · 2.15.2 Solving linear systems without inversion via diagonalization · 2.15.3 PyTorch code for solving linear systems via diagonalization · 2.15.4 Matrix powers using diagonalization · 2.16 Spectral decomposition of a symmetric matrix · 2.16.1 PyTorch code for the spectral decomposition of a matrix · 2.17 An application relevant to machine learning: Finding the axes of a hyperellipse · 2.17.1 PyTorch code for hyperellipses · 2.18 Summary

What you learn in chapter 2 →

Chapter 3: Classifiers and vector calculus

This chapter introduces error minimization for classifiers by using gradients to adjust model weights and biases toward desired outputs.

3.1 Geometrical view of image classification · 3.1.1 Input representation · 3.1.2 Classifiers as decision boundaries · 3.1.3 Modeling in a nutshell · 3.1.4 Sign of the surface function in binary classification · 3.2 Error, aka loss function · 3.3 Minimizing loss functions: Gradient vectors · 3.3.1 Gradients: A machine learning-centric introduction · 3.3.2 Level surface representation and loss minimization · 3.4 Local approximation for the loss function · 3.4.1 1D Taylor series recap · 3.4.2 Multidimensional Taylor series and the Hessian matrix · 3.5 PyTorch code for gradient descent, error minimization,and model training · 3.5.1 PyTorch code for linear models · 3.5.2 Autograd: PyTorch automatic gradient computation · 3.5.3 Nonlinear Models in PyTorch · 3.5.4 A linear model for the cat brain in PyTorch · 3.6 Convex and nonconvex functions, and global and local minima · 3.7 Convex sets and functions · 3.7.1 Convex sets · 3.7.2 Convex curves and surfaces · 3.7.3 Convexity and the Taylor series · 3.7.4 Examples of convex functions · 3.8 Summary

What you learn in chapter 3 →

Chapter 4: Linear algebraic tools in machine learning

This chapter introduction frames high-dimensional data as structured matrices whose feature-vector rows occupy nonuniform regions of feature space, motivating the linear-algebra tools studied next.

4.1 Distribution of feature data points and true dimensionality · 4.2 Quadratic forms and their minimization · 4.2.1 Minimizing quadratic forms · 4.2.2 Symmetric positive (semi)definite matrices · 4.3 Spectral and Frobenius norms of a matrix · 4.3.1 Spectral norms · 4.3.2 Frobenius norms · 4.4 Principal component analysis · 4.4.1 Direction of maximum spread · 4.4.2 PCA and dimensionality reduction · 4.4.3 PyTorch code: PCA and dimensionality reduction · 4.4.4 Limitations of PCA · 4.4.5 PCA and data compression · 4.5 Singular value decomposition · 4.5.1 Informal proof of the SVD theorem · 4.5.2 Proof of the SVD theorem · 4.5.3 Applying SVD: PCA computation · 4.5.4 Applying SVD: Solving arbitrary linear systems · 4.5.5 Rank of a matrix · 4.5.6 PyTorch code for solving linear systems with SVD · 4.5.7 PyTorch code for PCA computation via SVD · 4.5.8 Applying SVD: Best low-rank approximation of a matrix · 4.6 Machine learning application: Document retrieval · 4.6.1 Using TF-IDF and cosine similarity · 4.6.2 Latent semantic analysis · 4.6.3 PyTorch code to perform LSA · 4.6.4 PyTorch code to compute LSA and SVD on a large dataset · 4.7 Summary

What you learn in chapter 4 →

Chapter 5: Probability distributions in machine learning

This introduction motivates probability distributions as tools for modeling uncertain, high-dimensional machine-learning data and previews the chapter’s probabilistic and PyTorch-based approach.

5.1 Probability: The classical frequentist view · 5.1.1 Random variables · 5.1.2 Population histograms · 5.2 Probability distributions · 5.3 Basic concepts of probability theory · 5.3.1 Probabilities of impossible and certain events · 5.3.2 Exhaustive and mutually exclusive events · 5.3.3 Independent events · 5.4 Joint probabilities and their distributions · 5.4.1 Marginal probabilities · 5.4.2 Dependent events and their joint probability distribution · 5.5 Geometrical view: Sample point distributions for dependent and independent variables · 5.6 Continuous random variables and probability density · 5.7 Properties of distributions: Expected value, variance, and covariance · 5.7.1 Expected value (aka mean) · 5.7.2 Variance, covariance, and standard deviation · 5.8 Sampling from a distribution · 5.9 Some famous probability distributions · 5.9.1 Uniform random distributions · 5.9.2 Gaussian (normal) distribution · 5.9.3 Binomial distribution · 5.9.4 Multinomial distribution · 5.9.5 Bernoulli distribution · 5.9.6 Categorical distribution and one-hot vectors · 5.10 Summary

What you learn in chapter 5 →

Chapter 6: Bayesian tools for machine learning

This chapter introduction contrasts Bayesian and frequentist views and frames unsupervised learning as fitting probability distributions to patterns in unlabeled data.

6.1 Conditional probability and Bayes’ theorem · 6.1.1 Joint and marginal probability revisited · 6.1.2 Conditional probability · 6.1.3 Bayes’ theorem · 6.2 Entropy · 6.2.1 Geometrical intuition for entropy · 6.2.2 Entropy of Gaussians · 6.3 Cross-entropy · 6.4 KL divergence · 6.4.1 KLD between Gaussians · 6.5 Conditional entropy · 6.5.1 Chain rule of conditional entropy · 6.6 Model parameter estimation · 6.6.1 Likelihood, evidence, and posterior and prior probabilities · 6.6.2 Maximum likelihood parameter estimation (MLE) · 6.6.3 Maximum a posteriori (MAP) parameter estimation and regularization · 6.7 Latent variables and evidence maximization · 6.8 Maximum likelihood parameter estimation for Gaussians · 6.8.1 Python PyTorch code for maximum likelihood estimation · 6.8.2 Python PyTorch code for maximum likelihood estimation using gradient descent · 6.9 Gaussian mixture models · 6.9.1 Probability density function of the GMM · 6.9.2 Latent variables for class selection · 6.9.3 Classification via GMM · 6.9.4 Maximum likelihood estimation of GMM parameters (GMM fit) · 6.10 Summary

What you learn in chapter 6 →

Chapter 7: Function approximation: How neural networks model the world

This chapter introduces neural networks as a computing paradigm that represents real-world problems as functions and approximates those functions with network building blocks.

7.1 Neural networks: A 10,000-foot view · 7.2 Expressing real-world problems: Target functions · 7.2.1 Logical functions in real-world problems · 7.2.2 Classifier functions in real-world problems · 7.2.3 General functions in real-world problems · 7.3 The basic building block or neuron: The perceptron · 7.3.1 The Heaviside step function · 7.3.2 Hyperplanes · 7.3.3 Perceptrons and classification · 7.3.4 Modeling common logic gates with perceptrons · 7.4 Toward more expressive power: Multilayer perceptrons (MLPs) · 7.4.1 MLP for logical XOR · 7.5 Layered networks of perceptrons: MLPs or neural networks · 7.5.1 Layering · 7.5.2 Modeling logical functions with MLPs · 7.5.3 Cybenko’s universal approximation theorem · 7.5.4 MLPs for polygonal decision boundaries · 7.6 Summary

What you learn in chapter 7 →

Chapter 8: Training neural networks: Forward propagation and backpropagation

This chapter introduces supervised neural-network training as fitting MLP parameters from labeled input-output examples, then previews the mathematical tools used to do it.

8.1 Differentiable step-like functions · 8.1.1 Sigmoid function · 8.1.2 Tanh function · 8.2 Why layering? · 8.3 Linear layers · 8.3.1 Linear layers expressed as matrix-vector multiplication · 8.3.2 Forward propagation and grand output functions for an MLP of linear layers · 8.4 Training and backpropagation · 8.4.1 Loss and its minimization: Goal of training · 8.4.2 Loss surface and gradient descent · 8.4.3 Why a gradient provides the best direction for descent · 8.4.4 Gradient descent and local minima · 8.4.5 The backpropagation algorithm · 8.4.6 Putting it all together: Overall training algorithm · 8.5 Training a neural network in PyTorch · 8.6 Summary

What you learn in chapter 8 →

Chapter 9: Loss, optimization, and regularization

This introduction frames neural-network training as minimizing a loss by optimizing weights, then previews loss choices, optimization methods, regularization, overfitting, and dropout.

9.1 Loss functions · 9.1.1 Quantification and geometrical view of loss · 9.1.2 Regression loss · 9.1.3 Cross-entropy loss · 9.1.4 Binary cross-entropy loss for image and vector mismatches · 9.1.5 Softmax · 9.1.6 Softmax cross-entropy loss · 9.1.7 Focal loss · 9.1.8 Hinge loss · 9.2 Optimization · 9.2.1 Geometrical view of optimization · 9.2.2 Stochastic gradient descent and minibatches · 9.2.3 PyTorch code for SGD · 9.2.4 Momentum · 9.2.5 Geometric view: Constant loss contours, gradient descent, and momentum · 9.2.6 Nesterov accelerated gradients · 9.2.7 AdaGrad · 9.2.8 Root-mean-squared propagation · 9.2.9 Adam optimizer · 9.3 Regularization · 9.3.1 Minimum descriptor length: An Occam’s razor view of optimization · 9.3.2 L2 regularization · 9.3.3 L1 regularization · 9.3.4 Sparsity: L1 vs. L2 regularization · 9.3.5 Bayes’ theorem and the stochastic view of optimization · 9.3.6 Dropout · 9.4 Summary

What you learn in chapter 9 →

Chapter 10: Convolutions in neural networks

This introduction explains why convolutional layers analyze local image patterns with shared weights and how stacked, trainable convolutions form hierarchical visual features within neural networks.

10.1 One-dimensional convolution: Graphical and algebraical view · 10.1.1 Curve smoothing via 1D convolution · 10.1.2 Curve edge detection via 1D convolution · 10.1.3 One-dimensional convolution as matrix multiplication · 10.1.4 PyTorch- One-dimensional convolution with custom weights · 10.2 Convolution output size · 10.3 Two-dimensional convolution: Graphical and algebraic view · 10.3.1 Image smoothing via 2D convolution · 10.3.2 Image edge detection via 2D convolution · 10.3.3 PyTorch- 2D convolution with custom weights · 10.3.4 Two-dimensional convolution as matrix multiplication · 10.4 Three-dimensional convolution · 10.4.1 Video motion detection via 3D convolution · 10.4.2 PyTorch- Three-dimensional convolution with custom weights · 10.5 Transposed convolution or fractionally strided convolution · 10.5.1 Application of transposed convolution: Autoencoders and embeddings · 10.5.2 Transposed convolution output size · 10.5.3 Upsampling via transpose convolution · 10.6 Adding convolution layers to a neural network · 10.6.1 PyTorch- Adding convolution layers to a neural network · 10.7 Pooling · 10.8 Summary

What you learn in chapter 10 →

Chapter 11: Neural networks for image classification and object detection

This introduction distinguishes image classification from object detection, explains why CNNs replaced hand-crafted vision features, and previews the chapter’s deep-learning architectures and applications.

11.1 CNNs for image classification: LeNet · 11.1.1 PyTorch- Implementing LeNet for image classification on MNIST · 11.2 Toward deeper neural networks · 11.2.1 VGG (Visual Geometry Group) Net · 11.2.2 Inception: Network-in-network paradigm · 11.2.3 ResNet: Why stacking layers to add depth does not scale · 11.2.4 PyTorch Lightning · 11.3 Object detection: A brief history · 11.3.1 R-CNN · 11.3.2 Fast R-CNN · 11.3.3 Faster R-CNN · 11.4 Faster R-CNN: A deep dive · 11.4.1 Convolutional backbone · 11.4.2 Region proposal network · 11.4.3 Fast R-CNN · 11.4.4 Training the Faster R-CNN · 11.4.5 Other object-detection paradigms · 11.5 Summary

What you learn in chapter 11 →

Chapter 12: Manifolds, homeomorphism, and neural networks

This chapter briefly introduces geometric intuitions about manifolds and homeomorphism and their role in neural networks.

12.1 Manifolds · 12.1.1 Hausdorff property · 12.1.2 Second countable property · 12.2 Homeomorphism · 12.3 Neural networks and homeomorphism between manifolds · 12.4 Summary

What you learn in chapter 12 →

Chapter 13: Fully Bayes model parameter estimation

This introduction frames unsupervised modeling as fitting a probability density whose samples overlap the training-data cloud, enabling generation and probability evaluation of new examples.

13.1 Fully Bayes estimation: An informal introduction · 13.1.1 Parameter estimation and belief injection · 13.2 MLE for Gaussian parameter values recap) · 13.3 Fully Bayes parameter estimation: Gaussian, unknown mean, known precision · 13.4 Small and large volumes of training data, and strong and weak priors · 13.5 Conjugate priors · 13.6 Fully Bayes parameter estimation: Gaussian, unknown precision, known mean · 13.6.1 Estimating the precision parameter · 13.7 Fully Bayes parameter estimation: Gaussian, unknown mean, unknown precision · 13.7.1 Normal-gamma distribution · 13.7.2 Estimating the mean and precision parameters · 13.8 Example: Fully Bayesian inferencing · 13.8.1 Maximum likelihood estimation · 13.8.2 Bayesian inference · 13.9 Fully Bayes parameter estimation: Multivariate Gaussian, unknown mean, known precision · 13.10 Fully Bayes parameter estimation: Multivariate, unknown precision, known mean · 13.10.1 Wishart distribution · 13.10.2 Estimating precision · 13.11 Summary

What you learn in chapter 13 →

Chapter 14: Latent space and generative modeling, autoencoders, and variational autoencoders

This introduction defines latent vectors as learned, compact representations of observed inputs that retain goal-relevant information while discarding the rest.

14.1 Geometric view of latent spaces · 14.2 Generative classifiers · 14.3 Benefits and applications of latent-space modeling · 14.4 Linear latent space manifolds and PCA · 14.4.1 PyTorch code for dimensionality reduction using PCA · 14.5 Autoencoders · 14.5.1 Autoencoders and PCA · 14.6 Smoothness, continuity, and regularization of latent spaces · 14.7 Variational autoencoders · 14.7.1 Geometric overview of VAEs · 14.7.2 VAE training, losses, and inferencing · 14.7.3 VAEs and Bayes’ theorem · 14.7.4 Stochastic mapping leads to latent-space smoothness · 14.7.5 Direct minimization of the posterior requires prohibitively expensive normalization · 14.7.6 ELBO and VAEs · 14.7.7 Choice of prior: Zero-mean, unit-covariance Gaussian · 14.7.8 Reparameterization trick · 14.8 Summary

What you learn in chapter 14 →

Concepts explained in this book

embeddingAn ordered mathematical object representing a point or direction in a coordinate space.
labelThe manually labeled or authoritative target output for a training example against which prediction error is measured.
classifierA supervised learning task that assigns inputs to discrete classes.
objective functionA function assigning a scalar penalty to model parameters or predictions.
meanThe expected value or center parameter of a probability distribution, especially a Gaussian distribution.
convolutional layerA neural-network layer that applies convolution operations to an input, commonly used to extract local features from images.
normal distributionA Gaussian probability distribution.
cross entropy lossA classification loss that measures the mismatch between a predicted probability distribution over classes and the target distribution.
latent spaceA learned lower-dimensional representation of an input.
covariance matrixA matrix summarizing variances and covariances among dimensions.
training datasetA set of input examples collected from a distribution, which may or may not have corresponding labels or outputs.
trainingThe step of estimating model parameters so the model’s outputs on training inputs match the corresponding targets as closely as possible.
parameterThe tunable values of a model estimated from data.
arrayAn indexed sequence of values, here used to represent a signal processed by a convolution.
linear regressionA function that preserves addition and scalar multiplication between vector spaces, as in matrix multiplication.
prior probabilityThe distribution over parameters before observing data.
varianceA scalar measure of how dispersed a set of values is around their mean, used here as the quantity assumed known in one conjugate-prior case.
principal component analysisA multivariate statistical technique that finds orthogonal directions of maximal spread in data using covariance-matrix eigenvectors and eigenvalues.
convolution kernelA learned kernel applied in a convolution layer to detect local spatial patterns in images.
probability distributionA probability law over variable values.
posterior probabilityThe updated distribution over model parameters after combining prior and likelihood.
random variableA quantity whose possible values are associated with probabilities, used to model uncertain measurable attributes or event outcomes.
maximum a posteriori estimationThe parameter value that maximizes the posterior distribution given the observed data.
probability theoryA measure of how likely an event or value is.
backpropagationThe PyTorch method call that triggers backpropagation to compute gradients from the loss.
dimensionality reductionThe reduction of the effective number of dimensions in a representation by a transformation such as a singular weight matrix.
gradientThe vector of partial derivatives indicating the local direction of steepest increase of a function, used here to choose the steepest ascent or descent direction.
non-maximum suppressionA post-processing method that removes redundant overlapping detections while keeping the highest-scoring ones.
maximum likelihood estimationA parameter-estimation method that chooses values maximizing the likelihood of the observed data under a probabilistic model.
bayesian statisticsA parameter-estimation approach that combines observed data with a prior distribution to produce a posterior distribution over unknown quantities.
reconstruction lossThe discrepancy between an input and its reconstruction.
likelihoodA function of model parameters measuring how probable observed data are under those parameters.
Kullback–Leibler divergenceA loss term based on Kullback-Leibler divergence that penalizes differences between probability distributions, such as an encoder's latent distribution and a chosen prior.
Intersection over UnionA bounding-box overlap metric used to match proposed regions to ground-truth boxes.
regularizationThe practice of constraining a learned representation or model to encourage desirable structure such as smoothness or compactness.
entropyA measure of uncertainty or unpredictability in a probability distribution, high when outcomes are nearly equally likely and low when one outcome dominates.
Gaussian mixture modelA probabilistic model representing a distribution as a weighted combination of Gaussian components.
perceptronA simple linear threshold unit that computes a weighted sum of inputs.
max poolingMax pooling is an aggregation operation that replaces each local or grouped region with its maximum value, often to downsample a representation.
Region Proposal NetworkThe region proposal network component of Faster R-CNN that generates candidate object regions from convolutional features.
inner productA scalar computed from two vectors by multiplying corresponding components and summing them, used here to measure one vector's projection along another.
feature spaceThe set of all possible input values represented geometrically as the space in which data points lie.
projectionA transformation that maps points from a higher-dimensional space to a lower-dimensional one by discarding coordinates or components.
softmaxA smooth function that converts an unnormalized score vector into a probability-like distribution over classes.
autoencoderA neural network model trained to compress inputs into a latent representation and reconstruct the original inputs from that representation.
feature mapA spatial array of activation values produced by applying a convolution or pooling operation over an input image or prior layer output.
probability densityA function describing probability density for continuous-valued distributions.
singular value decompositionThe statement that any matrix can be decomposed as a product of orthogonal matrices and a diagonal matrix in singular value decomposition form.
expressive powerThe capacity of a model architecture to represent a wide range of functions or input-output relationships, including ones like XOR.
inputA measured or encoded property used as an input variable to a machine-learning model.
weighted sumA linear combination of values multiplied by coefficients.
model architectureThe chosen arrangement of neurons and their connections in a neural network.
RoI poolingA pooling operation that converts variable-sized proposed regions into a fixed-size representation for downstream detection layers.
strideThe step size with which a convolutional kernel moves across the input, affecting the output size in convolution and transposed convolution.
smooth l1 lossA piecewise regression loss that behaves like squared error near zero and absolute error for large residuals.
eigenvectorA nonzero vector that is scaled without changing direction by a linear transformation.
variational autoencoderAn autoencoder that models the latent space probabilistically and is trained with regularizing constraints.
anchor boxA preset bounding box template with a specific scale and aspect ratio used as a reference for object localization.
decodeThe reverse mapping that reconstructs data from a reduced representation.
Fast R-CNNAn object-detection architecture that computes convolutional features once per image and then performs region-wise classification and bounding-box regression.
machine learningA paradigm for learning a parameterized mapping from inputs to outputs by training on data rather than by writing explicit step-by-step rules.
decision boundaryA decision boundary that is a straight line or hyperplane, unable to separate the XOR points in this example.
linear layerA linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.
region proposalA candidate image region proposed as possibly containing an object for later classification and localization.
objectnessA predicted score indicating how likely an anchor or proposal contains an object rather than background.
sample point distributionA collection of sampled values viewed as points in the variable's domain, used to visualize the distribution they came from.
standard deviationThe square root of variance, used as a spread measure in the same units as the underlying values.
supervised learningA learning setting where training examples include known target outputs used to guide parameter estimation.
similarity searchThe task of finding documents relevant or similar to a query or another document using feature representations.
eigenvalueA scalar associated with a linear transformation that gives the scaling factor for a corresponding eigenvector.
manifoldA one-dimensional manifold.
mixture weightA nonnegative coefficient in a Gaussian mixture model that gives the prior proportion assigned to a component.
interceptA learned additive parameter that shifts model output independently of the inputs.
linear systemA set of linear equations considered together, typically represented in matrix form and solved for unknown variables.
overfittingA training failure mode in which a model fits idiosyncrasies of the training data too closely and generalizes poorly.
loss surfaceThe geometric representation of a loss function over parameter space, where height indicates loss value for each parameter setting.
vanishing gradient problemThe training difficulty in deep networks where repeated multiplication of small derivatives makes early-layer gradients nearly zero.
hyperplaneA flat boundary of one dimension less than the ambient space.
encoderThe neural-network component that maps an input to a latent-space distribution.
matrix-matrix multiplicationThe result of multiplying two compatible matrices, here discussed under the rule that transposition reverses the order of factors.
cosine similarityA dot product computed on unit-length vectors so that the score is insensitive to vector length.
bounding boxThe numeric values that specify the rectangular extent of a detected object within an image.
categorical encodingA vector with exactly one element equal to 1 and all others equal to 0, used here to encode the correct class.
LeNetA convolutional neural network architecture introduced for digit recognition and used here as the example image classifier.
joint probability distributionThe probability that a particular combination of values for two or more random variables occurs together.
quadratic formA homogeneous degree-two expression, here written with a symmetric matrix as x^T A x or equivalent forms describing geometric shapes and curvature.
ResNetA deep convolutional neural network architecture that uses identity shortcut connections to ease optimization of very deep networks.
generationA modeling approach that learns a data distribution so new samples resembling the training data can be generated.
stochastic gradient descentStochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
PyTorchAn open-source machine learning library used here to provide the chapter's executable code examples and automatic gradient computation.
nonlinear mappingA property of a function or layer whose output is not a linear transformation of its inputs.
quantizationThe conversion of floating-point region boundaries or ratios into integer-aligned indices for feature extraction.
object detection modelA model that predicts object classes and their spatial locations in an image.
principal componentThe projection values of data onto a principal axis in PCA, representing coordinates along that direction.
gradient descentGradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
independenceA variable that is statistically unrelated to another variable, with their joint probability equal to the product of their marginal probabilities.
residual connectionA connection that adds a layer's input to its output so information and gradients can bypass one or more layers.
VGGA family of convolutional neural networks characterized by repeated 3×3 convolutions and deep stacked blocks for image classification.
fully bayes parameter estimationA Bayesian parameter-estimation approach that combines observed data likelihood with a prior distribution to obtain a posterior over the unknown parameter.
paddingThe treatment of positions where a convolution kernel extends beyond the input array, such as stopping early or pretending missing values are zero.
mean squared errorThe average of squared differences between predicted and target values.
deltaAn intermediate variable used in backpropagation to represent the rate of change of loss with respect to a layer's pre-activation output.
neuronA biological nerve cell used as an analogy for a neural-network building block.
poolingA downsampling operation on feature maps that reduces spatial resolution, often by local averaging or max selection.
Universal Approximation TheoremA universal-approximation result stating that a sufficiently wide single-hidden-layer neural network can approximate continuous functions on compact domains arbitrarily well.
parameterized functionA function whose behavior depends on adjustable parameter values learned from data.
transposed convolutionA convolution-like operation often described via the transpose of the convolution matrix, used here as a motivation for understanding the matrix form of convolution.
data transformationA mapping between geometric spaces, used here as the view of a machine learning model as mapping input points to output points.
binary random variableA variable restricted to two possible values, used here to encode yes/no logical states in machine inputs and outputs.
sum ruleThe probability rule stating that the total probability over all possible outcomes equals one.
displacement vectorA vector describing a change in position or parameters, such as the step taken in parameter space.
partial derivativeThe derivative of a multivariable function with respect to one variable while holding the others constant, here squared and accumulated by AdaGrad.
reconstructed imageThe output produced by an autoencoder that approximates the original input after decoding from a latent representation.
similarity metricA measure of how alike two represented items are, especially when computed from their vector representations.
learning rateThe step-size setting that controls how far model parameters move on each optimization update, possibly chosen separately for each parameter dimension.
binomial distributionA discrete probability distribution giving the chance of exactly k successes in n independent trials with constant success probability π.
dropoutA regularization method that randomly turns off units during training to reduce co-adaptation and improve generalization.
q(z|x)The encoder-defined approximate posterior distribution over latent variables for a given input x in a variational autoencoder.
pseudo-inverseA pseudoinverse is a generalized matrix inverse that provides least-squares solutions when an ordinary inverse does not exist.
convolutional neural networkA neural network that uses convolutional layers to process structured grid data such as images.
principal valueA scalar associated with a principal axis in this PCA discussion, where smaller values indicate directions that can be discarded with little information loss.
evidence lower boundThe variational objective optimized in a VAE, combining reconstruction quality with a regularization term on the latent distribution.
pointwise convolutionA convolution with a one-by-one kernel that mixes channels at each spatial location and can reduce dimensionality.
forward passThe layer-by-layer computation, or forward pass, that produces neural-network activations and outputs from an input.
precisionThe precision parameter in the normal-gamma distribution.
unsupervised learningA learning setup that uses training inputs without manually provided target outputs.
expectation maximizationAn iterative algorithm for fitting latent-variable models by alternating between computing posterior responsibilities and updating parameters.
marginal probabilityThe probability distribution for one variable obtained by summing out the other variables from a joint distribution.
rotation matrixA matrix that represents a geometric rotation in this section, with orthogonal rotation matrices being length-preserving and invertible by transposition.
degradationIn residual networks, the phenomenon where adding more layers makes training and test performance worse instead of better.
automatic differentiationA method for mechanically computing derivatives of functions by tracing elementary operations.
unit vectorA vector whose length is exactly 1, used here to restrict the optimization to direction rather than magnitude.
momentumAn optimization technique that averages current and past gradients to smooth noisy updates and speed movement through flat regions.
kernel sizeThe spatial extent of the sliding window used by a pooling layer, determining the patch size summarized at each step.
Faster R-CNNAn object-detection model that combines a region proposal network, RoI pooling, and classification/regression heads to detect and localize objects in images.
target functionThe unknown function expressing desired input-output behavior that a model aims to approximate.
conditional probabilityThe probability of an event computed under the assumption that another event or condition is already known to hold.
momentumAn optimization strategy that accelerates parameter updates using accumulated past gradients or a lookahead variant to smooth descent on the loss surface.
function approximationThe task of representing a target input-output mapping with a model, used here to justify nonlinear models for complex real-world processes.
Huffman encodingA prefix-coding method that builds variable-length codes from symbol frequencies to compress data efficiently.
variable bit-rate codingA coding scheme that assigns shorter codes to more probable symbols and longer codes to less probable ones to reduce average storage cost.
neural network layerA stage in a neural network that transforms one set of activations into another.
gamma distributionA probability distribution over positive continuous values, here mentioned as the univariate distribution previously used in Bayesian parameter estimation.
Wishart distributionThe space of symmetric positive semidefinite matrices that Wishart-distributed random matrices range over.
convexityThe property of a function or set that, in this section, is characterized by being above its tangent and by a nonnegative second derivative.
l2 normThe Euclidean norm measuring a vector's length by squaring components, summing them, and taking the square root.
hinge lossA classification loss that is zero once a margin criterion is satisfied and increases only when incorrect classes score too close to or above the correct class.
logitA vector of unnormalized class scores produced by a classifier before conversion to probabilities.
matrix inverseFor a square matrix, the matrix that undoes multiplication by the original matrix when it exists.
logical XORA Boolean operation that outputs true when exactly one of its two binary inputs is true.
Nesterov momentumA momentum-based optimization method that evaluates the gradient at a lookahead point before combining it with past updates.
normal-gamma distributionA joint probability distribution over a mean parameter and a precision parameter, defined here as the product of a normal distribution and a gamma distribution.
AdaGradAn adaptive gradient optimization method that decreases the effective learning rate for parameters that have received large historical gradients.
principal axisAn axis in PCA aligned with a principal component, used as the direction along which data spread is measured.
spatio-temporal volumeA three-dimensional tensor representing video data across space and time.
Taylor seriesA multivariable Taylor expansion that approximates a function near a point and yields quadratic terms used in optimization arguments.
convolutional blockA repeated network unit composed of one or more convolution layers and a pooling step in the VGG architecture.
ReLUThe rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
mutually exclusive eventEvents that cannot occur together because their pairwise intersections are empty.
principal vectorAn eigenvector in PCA that indicates a direction of variation in the data, paired with a principal value.
categorical dataA variable that takes one value from a finite set of categories rather than a continuum of real values.
transpositionThe operation of swapping a matrix’s rows and columns to produce its transpose.
reconstructionThe process of producing an output intended to match the original input from a compressed representation.
term frequency-inverse document frequencyA document-representation weighting scheme that combines term frequency with inverse document frequency to emphasize discriminative words.
binary outcomeA trial result with exactly two possible states, here interpreted as success versus failure.
adamAn adaptive optimization algorithm that combines momentum-like first-moment estimates with per-parameter learning-rate adaptation.
forwardThe PyTorch module method that defines how inputs are transformed into outputs during a forward pass.
torch.nn.ModuleThe PyTorch base class that neural network models subclass to define learnable modules and a forward pass.
torch.nn.SequentialThe PyTorch container that chains modules in order so the output of one becomes the input of the next.
closed formA solution method that computes parameter values directly by algebraic manipulation rather than by iterative optimization.
pixelA position in a digital image whose stored value corresponds to local brightness or color, used here across time in video frames.
GoogleNetA deep convolutional neural network architecture that improved on earlier image-classification results.
high-dimensional dataA dataset whose instances are represented with many dimensions or features, making it suitable for dimensionality-reduction methods.
boolean operationA boolean-valued function over logical inputs, such as AND, OR, NOT, or XOR, used here as an example family of functions that can be emulated by perceptrons or MLPs.
focal lossA modified classification loss that downweights easy examples and emphasizes hard examples, especially under class imbalance.
S / ATA / A^T A / A A^TThe matrix product of A transpose and A, used here as the matrix whose eigenvalues are the singular values mentioned in the section.
conjugacyA prior distribution chosen so that, after Bayes' rule is applied with a specified likelihood family, the posterior stays in the same distribution family.
sigmoidAn S-shaped nonlinear activation function whose steepness and horizontal position can be adjusted with parameters.
demand predictionThe task of estimating future product demand from historical and contextual variables.
min-max normalizationA preprocessing method that rescales values into a fixed range, here 0–1.
document-term matrixA matrix with documents as rows and terms as columns, storing term-frequency values for retrieval or analysis.
semantic similaritySimilarity of meaning between terms or documents, as opposed to exact string overlap.
topicA latent group of semantically related terms represented as a direction or subspace in document feature space.
topic spaceThe reduced vector space whose coordinates are topic dimensions rather than original terms.
topic vectorA vector whose entries indicate how strongly each term contributes to a topic.
latent semantic analysisA document-retrieval technique that uses low-dimensional structure in document-term data to capture latent topics and relationships.
output spaceThe space of all possible model outputs, treated geometrically as points the model maps to.
log probabilityThe natural logarithm of a probability density or mass value, used to make products and tiny probabilities easier to compute with.
valid paddingA convolution padding scheme that adds no extra border values around the input, so the kernel only applies where it fully fits inside the input.
diagonal matrixA square matrix whose only possibly nonzero entries are on the main diagonal.
law of large numbersA probabilistic principle stating that sample averages converge toward the expected value as the number of samples grows.
locally EuclideanA property of a space where each point has a neighborhood that can be mapped to Euclidean space without distortion of local topology.
exhaustive event setA set of events whose union covers the entire sample space so that one of them must occur.
p(|)The true target posterior distribution over latent variables that the variational approximation tries to match.
frequentist paradigmA probability interpretation that defines probability as relative frequency within a population or repeated trials under a specified condition.
optimizationA training component that updates model parameters according to a chosen optimization rule such as SGD.
singular valueA nonnegative quantity from singular value decomposition that measures how much a matrix stretches along an orthogonal direction.
hypersurfaceA surface of arbitrary dimension.
sign functionThe positive, negative, or zero indicator of a numeric expression, used here to determine which side of a decision boundary a point lies on.
matrix-vector multiplicationThe multiplication of a matrix by a compatible vector to produce a vector; here it is the algebraic form used to express convolution layers.
channelA channel dimension in a tensor that separates different input feature maps or image channels.
homeomorphismA topology-preserving mapping between manifolds that allows stretching and squishing but not tearing, breaking, or folding.
diagonalizationA matrix transformation technique that expresses a square matrix in a basis where it becomes diagonal, simplifying computations such as powers.
categorical distributionA discrete probability distribution over exactly one of several mutually exclusive classes in a single trial.
orthogonalityPerpendicular in the geometric sense; here used for vectors or matrices whose directions are mutually independent or right-angled.
multinomial distributionA probability distribution over counts of outcomes from repeated independent trials with multiple mutually exclusive categories.
determinantA scalar quantity computed from a square matrix, appearing here in the Wishart density via matrix absolute-value notation.
coordinate systemA coordinate system with perpendicular axes used here as the original input-space representation before latent transformation.
linear dependenceA property of a set of vectors where at least one vector can be written as a linear combination of the others.
level contourA curve in a two-dimensional domain along which a function has the same value everywhere.
level surfaceA level surface in an arbitrary number of dimensions, generalizing constant-value contours and surfaces.
dimensionThe number of independent coordinates in an input space, used here to distinguish one-dimensional from two-dimensional constructions.
edgeA boundary-like location in an input array where values change abruptly, producing a strong convolution response here.
tanhA hyperbolic tangent activation function used to introduce nonlinearity in neural networks.
auxiliary classifierAn extra intermediate classifier attached to a deep network to provide additional training signals to earlier layers.
auxiliary lossAn additional loss term computed from an intermediate network output to help optimize a deep model.
InceptionA convolutional neural-network architecture that increases depth and width using parallel multi-scale branches while keeping computation manageable.
Inception moduleA multi-branch convolutional network block that increases a network's depth and width efficiently.
network-in-network paradigmA design pattern that replaces a single large convolution with multiple smaller and parallel transformations inside a network block.
zero paddingA convolution setting that extends the input with zeros so the kernel can produce outputs near the boundary.
logical AND operatorA binary logical operation that outputs true only when both inputs are true.
logical NOTA unary logical operation that inverts a binary input value.
logical ORA binary logical operation that outputs true when at least one input is true.
basisA vector in a spanning set that provides a coordinate direction for representing every vector in the space.
object detectionA computer-vision task of identifying object categories in an image and estimating their spatial locations, often with bounding boxes.
local averagingA smoothing technique that reduces short-scale fluctuations by averaging nearby values in one dimension.
local patternA pattern that is confined to a small neighborhood of the input and can be captured by a local operator such as convolution.
tangent planeA line or direction that touches a curve locally and indicates the direction along which movement leaves a level contour unchanged.
hyperellipseA generalization of an ellipse to higher dimensions.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.