liveBookmanning.com

What you will learn in Deep Learning with PyTorch, Second Edition

Training and applying deep learning and generative AI models

by Howard Huang, Eli Stevens, Luca Antiga, Thomas Viehmann

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• trainingThe optimization phase in which a model learns from examples.
• language modelA language model with enough scale to generate and answer with fluent text from prompts.
• parameterA learnable numeric value in a model.
• arrayA container for indexed collections of values.
• objective functionLoss computed on training data during fitting.
• convolutional layerA PyTorch 2D convolution layer module that applies learned filters over spatial input regions.
• image segmentationA learning task that partitions an input, often an image, into labeled regions or elements, used here to flag voxels or pixels that may belong to a nodule.
• linear layerA PyTorch linear layer module that applies an affine transformation to map features to outputs.
• datasetA collection of examples and annotations used for model training or evaluation.
• DataLoaderA PyTorch abstraction that batches and loads samples from datasets.
• torch.nn.ModuleThe PyTorch base class for defining neural-network models as Python modules with parameters, submodules, and a forward method.
• overfittingA failure mode where a model memorizes irrelevant details of the training data and generalizes poorly, discussed here as a risk of increased capacity.
• maskMasking is the selective representation, concealment, inclusion, exclusion, or restriction of parts of data or a system according to specified criteria.
• categorical encodingAn encoding in which exactly one position is set to indicate a category and all others are zero.
• embeddingA one-dimensional numeric data structure used here as a mathematical object on which sums and dot products can be computed.
• PyTorchThe PyTorch core Python library namespace that exposes tensor operations and functions such as mathematical elementwise operations.
• batch sizeThe number of training examples processed together in one update step.
• data augmentationA training technique that creates modified versions of examples to increase effective dataset diversity while preserving task-relevant meaning.
• lung noduleA small tumor in the lung, often only a few millimeters wide, that may later be classified as malignant or not.
• softmaxA normalization function that converts a vector of scores into a probability distribution by exponentiating and renormalizing the values.
• gradient descentGradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• fine-tuningThe process of adapting a pretrained model to a new task by continuing training on task-specific data.
• foundational modelA neural network already trained on prior data and reused for inference or further training.
• torch.compile()torch.compile() is a PyTorch API that compiles functions or modules into optimized executable code for execution.
• convolution kernelA small learned filter window applied over local regions of an input to compute weighted sums.
• SAMA transformer-based image segmentation model designed to generalize to many objects and support prompt-driven, zero-shot mask prediction.
• generationThe task of producing a token sequence as output, conditioned on preceding input or context.
• noiseNoise is variation, content, or data treated as unwanted, irrelevant, or distinct from the signal, pattern, or structure of interest.
• thresholdA cutoff value used here to turn a continuous wine-quality-related variable into a binary prediction.
• diffusion modelGenerative models that create outputs by reversing a diffusion process, mentioned here as part of the Hugging Face model collection.
• transformer architectureA neural network architecture for sequence modeling and content understanding based on attention rather than recurrence or convolution.
• batch normalizationA neural-network layer or operation that normalizes activations using batch statistics and behaves differently in training versus evaluation.
• backpropagationA PyTorch autograd call that performs reverse-mode differentiation.
• optimizer stateThe internal state of an optimization algorithm, such as momentum buffers, that can be saved to resume training exactly.
• linear regressionA mathematical mapping that combines inputs linearly, used here as a component of neural network layers before activation.
• activation functionA nonlinear function applied within a neural network to transform signals so the whole model can represent more complex mappings than a purely linear model.
• attentionA neural-network technique that helps a model focus on the most relevant parts of its input when computing representations.
• sequenceA dataset whose observations are ordered successive timepoints.
• max poolingMax pooling is an aggregation operation that replaces each local or grouped region with its maximum value, often to downsample a representation.
• translation invarianceThe property of a model or representation whose output is unchanged, or changes minimally, when an input object shifts position.
• torch.nnPyTorch’s neural-network module namespace containing classes for layers, containers, and other building blocks.
• cross entropy lossA loss value based on cross-entropy, used here as the quantity minimized during training.
• generalizationThe ability of a trained model to perform well on data not seen during training by capturing underlying patterns rather than memorizing specifics.
• residual connectionA connection that bypasses one or more network layers to provide an alternate path for activations or gradients.
• unsqueezeA tensor operation that inserts a new axis of size 1 at a specified position without changing the underlying data.
• convolutional neural networkA neural network that uses convolutional layers to process spatial data.
• labelA target label restricted to one of two classes, used here in binary classification and metric computation.
• computed tomographyComputed tomography imaging data consisting of cross-sectional body slices.
• automatic differentiationA technique for automatically computing derivatives needed for training neural networks.
• recallA classification metric measuring the fraction of actual positives that are correctly identified, which can be undefined when there are no actual positives.
• broadcastingA shape-alignment rule for element-wise operations that lets arrays or tensors of different shapes work together when dimensions are compatible.
• evaluation datasetHeld-out data used to assess generalization during model development.
• precisionNumerical precision is the fineness with which a numerical representation or computation distinguishes and retains values.
• regularizationA set of techniques that reduce overfitting and improve generalization by constraining model behavior during training.
• weight decayAn optimization method that shrinks parameters in proportion to their current values, generally distinct from L2 regularization for adaptive optimizers.
• training datasetThe subset of data used to fit a model.
• viewA tensor representation that shares underlying storage while presenting the same data with a different shape or interpretation.
• f-scoreA harmonic-mean classification metric that balances precision and recall, with the F-score parameter changing the relative emphasis between them.
• torch.distributedPyTorch’s distributed-training submodule for coordinating computation and communication across processes, devices, or machines.
• data parallelA training strategy that replicates a model across devices while splitting data among them.
• torch.exportA PyTorch API for exporting models for deployment within the PyTorch ecosystem, including ahead-of-time compilation.
• no_grad()A PyTorch context manager that disables gradient tracking during inference to reduce memory use and overhead.
• tensor shapeThe sizes of an array or tensor along each axis, here specifically the channels-height-width layout for image tensors.
• receptive fieldThe region of the original input that can influence a particular output unit in a convolutional network.
• deploymentThe process of putting a trained model into a real runtime environment where it serves users or applications.
• gradientA vector of derivatives describing a function’s change with respect to its inputs or parameters.
• learning rateThe step-size hyperparameter that controls how much model parameters are changed per update during optimization.
• computation graphA graph representation captured from model execution that records operations and data flow.
• model parallelismA strategy that places different parts of a model on different devices.
• tensor storageA one-dimensional memory buffer underlying tensor views.
• classifierA supervised learning task that assigns a text sequence to one of several categories.
• positional embeddingAn embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
• data representationAn internal data encoding produced or consumed by a model.
• TensorBoardA visualization tool for inspecting machine-learning training runs, metrics, and related summaries in a browser.
• normalizationNormalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.
• class balancingA training-data preparation technique that equalizes class frequencies so the model does not learn a degenerate majority-class shortcut.
• discriminationThe ability of a model to separate two classes from each other.
• strideTensor metadata describing its shape and how elements are laid out and indexed in storage, including step size between positions.
• logitUnnormalized model scores produced before applying a probability normalization such as softmax, used here as the direct input to cross-entropy loss.
• to methodThe PyTorch tensor method used to move tensors and model parameters to a specified device or change their type.
• distributed trainingA training approach that distributes computation or data across devices or machines.
• biasA learned additive parameter that shifts a neural-network layer’s output.
• Vision TransformerA transformer-based neural network adapted for computer vision tasks, especially image classification, by processing image patches with attention.
• vanishing gradient problemThe difficulty in training deep networks when gradients become too small as they propagate backward through many layers.
• class imbalanceA dataset in which class frequencies are heavily skewed toward one class.
• LogSoftmaxThe logarithm of the softmax normalization used to produce log-probabilities for classification.
• negative log-likelihoodA loss function that penalizes low predicted probability assigned to the correct class by taking the negative logarithm of that probability and summing over samples.
• cancerThe property of a tumor being cancerous and capable of harmful growth, as distinguished from benign tissue.
• transpositionAn operation that rearranges the dimensions of a tensor by swapping selected axes, changing its shape and stride without changing its values.
• floating-point arithmeticNumeric values represented approximately in binary floating-point form, used here as the tensor element type for loaded data.
• encodingA tokenizer function that converts text into numeric token representations.
• dropoutA regularization technique that randomly omits activations during training and is turned off or altered for inference.
• false positiveNegative cases that a classifier incorrectly predicts as positive, increasing the apparent success on the positive class but hurting utility.
• streamingSending generated output incrementally to a client as it becomes available rather than waiting for the full result to finish.
• torch.fxA graph representation produced by TorchDynamo that stores recorded instructions for later optimization or export.
• PyTorch Dynamo compilerA PyTorch runtime graph-capture tool that intercepts Python execution to trace and optimize computation graphs.
• data localityThe tendency for nearby memory accesses to hit nearby stored data, which improves performance.
• depthThe extent to which a neural network has many stacked layers or sequential operations, increasing representational complexity.
• ResNetA family of residual neural network architectures that use skip connections to ease training of deep models.
• categorical dataA variable type whose values are treated as discrete labels rather than ordered magnitudes.
• expressive powerA model's ability to fit a wide range of functions or memorize patterns, often increasing with parameter count and effective use of parameters.
• denoisingThe process of removing noise from a corrupted input, here used iteratively in a diffusion model to reconstruct an image.
• decodeA tokenizer function that converts numeric token representations back into text.
• pipelineA pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
• inpaintingAn image-generation/editing technique that fills in masked regions of an input image using a model and conditioning prompt.
• torch.distributed.init_process_groupThe PyTorch distributed API call that initializes a process group for collective communication.
• validation lossLoss computed on held-out validation data.
• L1 regularizationA regularization method that adds the sum of absolute weight values to the objective, encouraging sparse parameters.
• torch.onnx.exportA PyTorch export API that writes a model into ONNX format, optionally using Dynamo-based capture.
• mean squared errorA loss function that measures the average of squared differences between predicted and target values.
• scalarA single numeric value, contrasted here with multi-dimensional feature representations.
• indexingSelecting tensor elements or slices by using tensor values as indices or masks.
• ExecuTorchA PyTorch ecosystem library/runtime for deploying models efficiently on constrained devices such as mobile and embedded systems.
• model.parameters()A PyTorch module method that returns a module’s learnable parameters.
• pipeline parallelismA model-parallel technique that splits a model into stages and overlaps their execution.
• machine learningA class of methods in which systems learn patterns from data to perform tasks without being explicitly programmed with task rules.
• SegformerA transformer-based semantic segmentation model designed to be lightweight and fine-tunable for specific datasets.
• epochOne complete pass through the entire training dataset used in training, after which each sample has been used once.
• eager evaluationAn execution model in which operations run immediately rather than being first built into a static graph.
• transformsThe torchvision submodule containing image transformation utilities used for preprocessing and augmentation.
• quantizationA model optimization technique that reduces numerical precision of parameters and operations to lower memory use and often speed up inference.
• tokenA text unit produced by tokenization, which may be a character, word, or subword depending on the tokenizer design.
• mutationA method call that modifies an existing tensor directly rather than creating a separate result tensor.
• ONNXA vendor-neutral standard and file format for representing neural network computations so models can be executed in different runtimes and devices.
• keyThe attention component that represents what a position contains and can be matched against a query.
• queryA learned projection of an input position used to score how relevant other positions are in attention.
• valueThe transformed representation vectors that are combined using attention weights to produce the output for each position.
• voxelA three-dimensional volume element used to discretize volumetric data.
• promptUser-provided guidance such as points, boxes, or masks that steers the segmentation model toward a desired object or region.
• structured dataData organized as rows and columns where each row is treated as an independent record without inherent temporal order.
• noise scheduleA diffusion-model component that controls the sequence of denoising steps during generation.
• nonlinear mappingA process whose behavior cannot be represented as a simple linear relation between inputs and outputs.
• full graph captureA compilation analysis mode that records an entire model computation as one graph instead of fragmenting it into pieces.
• ReLUA rectified linear activation function that outputs zero for negative inputs and the input itself for positive inputs.
• CIFAR-10A benchmark dataset of labeled natural images.
• forwardThe standard module-computation entry point in PyTorch, invoked by __call__ and used for the main forward pass.
• stochastic gradient descentAn optimization method that updates parameters using stochastic gradient descent.
• allreduceA collective operation that combines values across participants and distributes the result to all of them.
• process groupThe set of processes participating in a coordinated distributed-training run and sharing communication context.
• preprocessingData preparation performed before model execution, here separating image and prompt encoding from the main model run.
• mirrorAn augmentation operation that flips an input to create a reflected variant.
• rotationA data augmentation transform that turns an image or volume around an axis, here constrained to the X-Y plane for CT data.
• shiftA data augmentation transform that translates an image or volume by a spatial offset, often with resampling at non-integer positions.
• vocabularyThe set of distinct symbols or words a language model can assign probabilities to in this character- or word-based setting.
• tensor parallelismA technique that partitions layer parameters or computation across devices.
• GPT modelA decoder-only transformer architecture used for autoregressive text generation without a separate encoder stack.
• collective communicationA distributed communication pattern in which multiple workers exchange or synchronize data.
• moduleA reusable network substructure made of a sequence of layers.
• parallelismThe practice of dividing work into tasks that execute simultaneously.
• Hugging FaceA repository service for discovering, sharing, and exchanging pretrained models and weights.
• poolingA downsampling technique that reduces spatial resolution by aggregating local regions of a feature map.
• SequentialA container module that groups child modules in order, forming a sequential network structure.
• LUNA Grand ChallengeA lung nodule analysis dataset containing CT scan data and annotations used here as the source of raw medical imaging samples.
• cachingThe practice of storing preprocessed data temporarily so it can be reused efficiently later instead of recomputing it.
• SamAutomaticMaskGeneratorA convenience abstraction in the Segment Anything library that generates segmentation masks from a model and image.
• multi-head attentionAn attention mechanism that runs several attention heads in parallel, each learning different projections and relationships.
• visual encoderThe part of a multimodal model that converts an image into a numeric representation for downstream decoding.
• confusion matrixA 2D breakdown of classification outcomes into true/false positives/negatives used to analyze model predictions.
• AOTInductorPyTorch's ahead-of-time compiler that turns exported graphs into optimized executable code.
• derivativeA quantity describing how a function changes with respect to an input or parameter.
• image generationThe task of creating new visual content from learned patterns, mentioned here as a preceding example task.
• forward passThe stage of model execution that propagates inputs through the network to produce outputs and loss values.
• channelThe tensor axis reserved for channel information.
• __getitem__A Python special method that retrieves an item by index or key, used here by Dataset to return a sample and label.
• __len__A Python special method that returns an object's length for the len() built-in, used here by Dataset to report item count.
• model workerA background execution component that pulls requests from a queue and runs model inference on them.
• queueA first-in, first-out structure that stores incoming work items until a worker processes them.
• token generation loopAutoregressive generation that produces one token at a time and exposes each token as soon as it is created.
• DICOMA medical imaging standard for storing and exchanging scan data and associated metadata.
• betaPer-step noise schedule coefficients that control how much noise is added during diffusion.
• DeviceMeshA PyTorch abstraction for arranging devices into a communication mesh.
• n-dimensional parallelismA distributed-training approach combining multiple parallelism strategies along different dimensions.
• functionA mathematical mapping from inputs to outputs, used here as the target of approximation by learning systems.
• GradioA Python library for quickly building interactive web demos and prototypes around machine learning models.
• bigram modelA sequence model that predicts the next symbol from the immediately preceding symbol using pairwise transition probabilities.
• false negativeAn outcome that is predicted as not belonging to the positive class even though it is actually of interest or positive.
• nodule classification modelA model that assigns candidate lung nodules to classes such as nodule or non-nodule, and later malignant or benign.
• rankA participating worker or process in a distributed computation, often assigned a partition of the data or sequence.
• world sizeThe total number of participating processes in a distributed training job.
• evaluation modeA model setting that switches layers such as dropout or batch normalization into inference behavior.
• MPIA message-passing standard/backend used here for distributed training across multiple machines.
• functional APIPyTorch's stateless functional namespace providing operations that do not store learnable module state.
• kernel fusionAn optimization that combines multiple operations into fewer executable units to reduce overhead and improve speed.
• differentiable activationAn activation function that is differentiable so gradients can be computed for learning.
• sliceA two-dimensional cross-section extracted from a CT scan for segmentation training or visualization.
• universal approximatorA function class that can approximate a wide range of target functions to arbitrary accuracy under suitable conditions.
• chain ruleA differentiation rule for computing the derivative of a composite function by multiplying derivatives along the composition.
• lung tumor detectionThe task of automatically identifying suspected malignant lung tumors from chest CT scans.
• tensor operationAn operation provided by PyTorch that acts on tensor data, either as a torch-module function or as a tensor method.
• statistical generationA probabilistic model that generates text by learning sequence distributions from data and sampling from them.
• fine-tuning datasetA dataset assembled specifically to adapt a pretrained model to a new task or labeling scheme.
• segmentation maskA labeled image-like boolean map indicating which pixels belong to the target region in segmentation.
• torch.randnA PyTorch function that samples random values from a normal distribution into a tensor.
• fully sharded data parallelA distributed-training method that shards model state across devices to reduce memory use.
• decoder-only transformerA transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.
• just-in-time compilationA compilation strategy that performs code optimization or translation during execution rather than before runtime.
• coordinate system transformationThe mapping information used to convert nodule coordinates into array indices for the CT volume.
• torch::nn::ModuleThe C++ PyTorch base class for defining neural-network modules.
• data balancingAdjusting class proportions in a dataset to reduce skew and make learning less biased toward a dominant class.
• series UIDA unique DICOM identifier used to name and associate the files belonging to one CT scan series.
• compositionThe operation of applying one function to the output of another, used here to describe how layers are chained in a neural network.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.