liveBookmanning.com

What you will learn in Inside Deep Learning: Math, Algorithms, Models

Math, Algorithms, Models

by Edward Raff

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• attentionA technique that assigns weights to inputs and combines them so a model can focus on relevant information.
• linear layerA PyTorch module that applies an affine linear transformation to its input.
• residual connectionA bypass path in a residual network that routes information around one or more layers so signals or shapes can be preserved.
• foundational modelA model whose parameters were fitted on a source task or dataset before reuse.
• embeddingA one-dimensional array treated as a tensor and used as a model input or feature representation.
• transfer learningA training approach that adapts a pretrained model to a new task, usually by reusing learned weights and matching the original preprocessing expectations.
• cross entropy lossA differentiable classification loss that measures the mismatch between predicted and target probability distributions.
• discriminatorIn a GAN, the neural network that scores whether an example is real or generated.
• optimizationAn algorithm or object that updates model parameters to reduce a loss function.
• autoencoderA neural network trained to reconstruct its input, typically by compressing it into a lower-dimensional latent representation and decoding it back.
• recurrent neural networkA neural network that processes sequential data by reusing parameters across time steps and carrying a hidden state forward.
• latent spaceA learned internal encoding of input data intended to preserve information useful for later use or understanding.
• generatorIn a GAN, the neural network that maps latent inputs to synthetic examples.
• PyTorchA Python deep learning library that provides tensor computation, automatic differentiation, and data abstractions for neural network training.
• layer normalizationA normalization technique that standardizes activations within each example rather than across a batch.
• trainingThe process of fitting a model's parameters on data to improve performance on a task.
• no_grad()A PyTorch context manager that disables gradient tracking inside its block, commonly used for inference or evaluation steps.
• automatic differentiationA mechanism for computing derivatives of model computations automatically, used here to support training of networks with registered parameters.
• stateThe recurrent state vector carried by an RNN across time steps, used here to store sequence information and initialize decoding.
• generative adversarial networkA generative deep-learning model consisting of a generator and discriminator trained adversarially to produce realistic samples.
• Faster R-CNNAn object detection architecture that combines a backbone, a region proposal network, and RoI pooling to localize and classify objects in images.
• LSTMA gated recurrent neural-network architecture designed to learn long-range dependencies.
• adaptive poolingA pooling method that adjusts its pooling region to produce a requested output size, used here to collapse the time dimension.
• GRUA gated recurrent neural network architecture that updates a hidden state over sequence steps with reset and update gates.
• bounding boxA rectangular region defined by coordinates that encloses an object in an image for localization or annotation.
• pointwise convolutionA convolution with a 1-by-1 spatial kernel used to mix or change channel dimensions without expanding the spatial receptive field.
• shared weightA convolutional-network technique that reuses the same parameters at multiple input positions instead of learning separate parameters for each position.
• poolingA neural-network operation that downsamples local neighborhoods by aggregating their values.
• encoder-decoder architectureA neural sequence transduction model that maps an input sequence to an output sequence, potentially of different length, using an encoder and decoder.
• batch normalizationA normalization technique that standardizes activations using statistics computed across a batch.
• vocabularyThe set of distinct characters in the text corpus that is mapped to discrete symbols for the character-level model.
• convolution kernelA learned array of weights applied over local input regions to compute convolution outputs.
• learning rateThe step-size parameter controlling how far an optimization update moves in the direction suggested by the gradient.
• arrayAn array with any number of dimensions, including scalars, vectors, matrices, and higher-rank arrays.
• object detectionA vision task in which a system identifies and localizes multiple objects in a single image, often producing several predictions per image.
• nn.GRUCellA PyTorch recurrent-cell module that processes one time step at a time and maintains a hidden state.
• evaluation datasetA held-out dataset split used to evaluate or tune a model without using it for parameter learning.
• sigmoidA nonlinear activation function that maps real-valued inputs to a bounded interval.
• sequence classification problemA supervised learning task that assigns a class label to an input sequence.
• LeakyReLUA ReLU variant that keeps a small nonzero slope for negative inputs so gradients do not vanish entirely on that side.
• mode collapseA GAN failure mode in which the generator produces only a limited variety of outputs instead of covering the data distribution.
• transposed convolutionA convolution-like operation used to increase spatial resolution in a tensor, often to undo downsampling from pooling in dense prediction models.
• adamAn adaptive optimization algorithm that combines moment estimates with variance-based gradient normalization.
• aliasingA sampling artifact in which fine-grained detail is collapsed into a smaller representation, causing distinct inputs to become indistinguishable.
• bidirectional rnnA recurrent-network configuration that processes a sequence in both forward and backward directions.
• Region Proposal NetworkThe region proposal network that generates candidate object regions for Faster R-CNN.
• image segmentationA computer vision task that assigns a label to each pixel in an image so that object regions are separated from the background or each other.
• score functionA function that computes an attention result from queries, keys, and values using their similarities.
• collate functionA data-loading callback that assembles individual dataset items into a batch.
• ReLUA rectified linear unit activation function that outputs zero for negative inputs and the input itself for nonnegative inputs.
• activation functionA nonlinear function applied in a neural network to transform layer outputs and increase representational power.
• Fashion-MNISTAn image dataset of clothing items commonly used as a harder alternative to MNIST.
• hyperparameterUser-chosen settings such as layer counts or learning rate that control model architecture or training behavior rather than being learned.
• principal component analysisPrincipal component analysis, a linear dimensionality-reduction method used here to project data into a lower-dimensional space.
• ResNetA neural network architecture built from residual blocks with skip connections that help information and gradients propagate through deep networks.
• encoderThe part of a sequence model or autoencoder that transforms an input sequence into an internal representation for downstream decoding.
• convolutional neural networkA neural-network design that uses convolutional layers to exploit spatial structure in image data.
• translation invarianceA property of a model whose output is unchanged, or less changed, when the input is shifted in position.
• region proposalAn object-detection approach that first proposes candidate regions likely to contain objects before classifying them.
• multi-head attentionAn attention mechanism that uses multiple attention heads to attend to different contexts or aspects of the input in parallel.
• U-NetAn encoder-decoder neural network architecture for image segmentation that combines low-level and high-level features via skip connections.
• parameterA numerical value that controls a model's computation and may be selected or updated during learning.
• teacher forcingA training approach for sequence decoders that feeds the true previous token as the next input instead of the model's own prediction.
• DataLoaderA PyTorch utility that prepares batches of examples from training or testing data for iteration during model execution.
• structural priorAn assumption about the organization of the data or task that is built into the model architecture before seeing data.
• ReduceLROnPlateauA PyTorch learning-rate scheduler that reduces the learning rate when a monitored quantity stops improving.
• ReZeroA training trick for residual-style networks that gates residual branches with a learnable scalar initialized to zero so the network starts near linear and gradually becomes more expressive.
• gradientThe vector of partial derivatives of a function with respect to its variables or model parameters.
• Order propertyInformation that depends on the order or position of elements in a sequence.
• tanhA nonlinear activation function that maps real-valued inputs into the interval [-1,1].
• self-supervised learningA training setup that derives supervisory signals from the data itself rather than from human-provided labels.
• pretrainingA strategy of training a model on a source dataset before adapting it to a downstream task.
• nn.ModuleListA PyTorch container that stores submodules in a list so they are registered and iterated over as model components.
• momentumA stochastic gradient-based optimization method that uses an accumulated velocity term for parameter updates.
• MixUpA data augmentation and regularization technique that creates synthetic training examples by linearly mixing two inputs and their labels.
• OptunaA hyperparameter-optimization library that searches parameter spaces by running trials and using a model to guide new suggestions.
• concatenationThe operation of joining tensors along a chosen axis, requiring the other axes to have matching sizes.
• vanishing gradient problemA training instability in which gradients become too small for effective learning updates.
• cosine annealingA learning-rate schedule that changes the learning rate according to a cosine-shaped pattern over training.
• dot scoreAn attention scoring method that uses the dot product between a state vector and a context vector, scaled by vector dimension.
• denoising autoencoderAn autoencoder trained to reconstruct clean data from corrupted input, used here as the analogy for sequence-to-sequence learning over sequences.
• evaluation modeA model setting used during inference or inspection that disables training-specific behavior such as dropout-like effects.
• warm startA transfer-learning technique that initializes a model from pretrained weights and then continues training those weights on the new task.
• softmaxA function that turns a vector of scores into normalized weights over items or classes by exponentiating and renormalizing them.
• GPUA graphics processing unit used as a parallel compute device for computation.
• local averagingAn image-smoothing transformation that averages local neighborhoods to reduce high-frequency detail before downsampling.
• autoregressive datasetA dataset construction for next-character prediction in which each input chunk is paired with the same chunk shifted by one position as labels.
• accuracyThe fraction of predictions that match the true labels.
• transformer architectureA sequence model built from positional encoding and multiheaded attention, used here as the standard original design for NLP and other tasks.
• timestepA discrete position in a sequence or recurrent computation at which the model updates state.
• gradient clippingA training stabilization method that caps gradient values to a chosen threshold to prevent excessively large updates.
• SequentialA PyTorch container module that applies a sequence of submodules in order.
• conditional GANA GAN variant that conditions generation on additional information such as class labels.
• Nesterov momentumA momentum variant that first applies the accumulated velocity, then computes a gradient at the lookahead position to correct the update sooner.
• average embeddingsA sequence-processing technique that replaces a variable-length sequence of embeddings with their mean over the time dimension.
• data leakageA data-preparation failure in which label-related information incorrectly appears in features and inflates apparent model performance.
• neural network layerA neural-network component that transforms its inputs.
• AdamWA variant of Adam that corrects the original algorithm’s issue and is used as the default optimizer in this book.
• gradient penaltyA regularization term that constrains discriminator gradients during Wasserstein GAN training.
• WGAN-GPA Wasserstein GAN variant that adds a gradient penalty to stabilize discriminator training.
• ToTensorA torchvision transform that converts a PIL image into a PyTorch tensor with values scaled to a numeric range suitable for learning.
• generator lossThe generator objective in WGAN training, defined here as the negative discriminator score on generated samples.
• RoI poolingA pooling procedure that extracts fixed-size representations from variable-sized proposed regions so later layers can process them uniformly.
• CUDANVIDIA's GPU computing software stack used here as the compute interface for running PyTorch operations on a GPU.
• matrix-matrix multiplicationThe binary operation that multiplies two matrices to produce a new matrix, written with the @ operator in Python and used here for performance comparison.
• spatial priorAn inductive bias that nearby positions in an input, such as image pixels, are more related than distant ones.
• gatingA mechanism that modulates information flow by multiplying signals by gate values, typically in the range from zero to one.
• Wasserstein GANA GAN variant that uses a Wasserstein-based objective to improve training behavior and stability.
• MNIST datasetA classic handwritten-digit image dataset of 28×28 grayscale examples used here to illustrate image loading and tensor structure.
• data augmentationThe practice of creating additional training examples by applying label-preserving transformations to existing data.
• discriminator lossThe loss used to train the discriminator in a GAN, combining its performance on real and generated data.
• real vs fakeThe two target classes used in GAN classification, representing genuine data and generated data.
• ImageNetA large image classification dataset with 1,000 classes used here as the pretraining source domain for ResNet.
• hidden layerA network layer between input and output that transforms representations before later computation, here before attention is applied.
• digital imageData whose values are arranged to represent images, typically as spatial grids of pixels or channels.
• source datasetThe dataset domain on which a model is originally pretrained before being adapted to a new task.
• stratified samplingA sampling method that preserves class proportions across data splits or samples.
• unsupervised learningA learning setting in which models are trained without human-provided labels, often by exploiting structure in the data.
• epochOne complete pass over a dataset during training or evaluation.
• dropoutA regularization technique that randomly zeros activations or inputs during training to reduce reliance on any single feature or unit.
• Gaussian noiseRandom perturbation sampled from a Gaussian distribution and added to data or activations to simulate noise.
• tensor shapeThe dimensions of a tensor, used to describe its layout and reshaping operations.
• additive scoreAn attention scoring method that combines representations through an additive learned function.
• packed sequenceA PyTorch sequence container that stores variable-length batches in packed form for efficient recurrent processing.
• exploding gradientA training instability where gradient magnitude becomes very large, leading to excessively large parameter updates.
• structural spatial priorThe assumption that nearby pixels and their arrangement carry meaning that should be exploited by image models.
• torchvisionA PyTorch companion library for computer-vision data loading and transforms used here to access MNIST and convert images to tensors.
• CosineAnnealingLRA PyTorch learning-rate scheduler class that implements cosine annealing.
• encode/decode processThe process of mapping an input into a latent representation and then reconstructing it back into an output.
• ensembleA collection of multiple models whose predictions are combined, typically to improve accuracy or robustness.
• stochastic weight averagingA training technique that averages model parameters collected at multiple points during training, often from learning-rate-cycle minima.
• hyperparameter tuningThe process of searching over user-chosen model settings to find values that improve an objective score.
• anti-aliasingA smoothing approach that reduces aliasing artifacts during sampling or downsampling by filtering before spatial reduction.
• decoder layerA transformer block that uses encoder outputs in a second attention stage for sequence generation.
• Transformer encoder layerA transformer block that processes an input sequence with self-attention and a feedforward step, and can be stacked repeatedly.
• one-to-many problemA function-like relationship in which one input can correspond to multiple valid outputs.
• alphaA learned scalar coefficient that scales the residual branch in ReZero and is initialized to zero at the start of training.
• spatial correlationA dependency pattern where nearby positions in an input, such as neighboring pixels, tend to be related.
• strided convolutionA convolution performed with stride greater than one so it reduces spatial resolution instead of using a separate pooling layer.
• target domainThe dataset or domain on which a pretrained model is adapted and evaluated in transfer learning.
• class imbalanceA dataset condition in which some classes occur much more frequently than others.
• attention subnetworkA smaller subnetwork that produces a score for each input item and converts those scores into attention weights.
• weighted averageAn average in which each term is multiplied by a weight before summation.
• GPGPUThe use of graphics processing hardware to speed up computation.
• attention-based networkA neural network that incorporates attention to adapt its computation to the input by focusing on relevant items.
• context vectorA summary vector formed from input representations, here produced by averaging or attention over word embeddings.
• tensor reshapingAn operation that changes a tensor's shape without changing its total number of values, used here to turn a network output into an image-shaped tensor.
• kernel sizeThe spatial extent of a convolutional filter.
• binomial filterA smoothing kernel whose weights follow binomial coefficients, emphasizing the center more than the edges.
• cropA preprocessing operation that extracts a spatial subregion from an image, here used to create training and test variants for shift analysis.
• fully convolutional networkA neural network composed only of convolutional layers plus supporting layers such as nonlinearity and normalization, with no dense linear layers.
• carry stateThe LSTM memory state that carries longer-term information separately from the hidden state.
• variable-length sequenceInput data whose number of elements can vary across examples and may require padding before batch processing.
• nonlinear mappingA transformation inserted between linear steps to let a neural network represent functions that are not purely linear.
• expressive powerThe set of functions or behaviors that a model architecture can express.
• BERTA pretrained transformer-based language model used for language understanding and transfer learning.
• dimensionality reductionThe process of mapping data from a higher-dimensional space into a lower-dimensional representation while retaining important structure.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.