liveBookmanning.com

What you will learn in Deep Learning with Python, Third Edition

by François Chollet and Matthew Watson

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• language modelA language model trained on large text corpora that can generate human-sounding responses, mentioned here as an example of a system that may be modular but still lacks recombination ability.
• generalizationThe ability of a model or agent to perform well on novel situations different from those seen during training or prior exposure.
• embeddingAn ordered numeric representation used here to encode source tokens and target tokens for attention computations.
• transformer architectureA neural-network architecture based on attention mechanisms, used here as the core architecture for large language models.
• generationThe process of producing new text token by token from a trained language model, often by feeding previous outputs back as inputs.
• classifierA supervised learning task that assigns text inputs to one of several classes.
• recurrent neural networkA neural network architecture that processes sequences by feeding information from one step to the next.
• overfittingThe failure mode where performance improves on training data while degrading on unseen data because the model has adapted too specifically to the training set.
• foundational modelA model trained beforehand on large-scale data and then reused for downstream tasks or analysis without starting from scratch.
• trainingThe phase during which a model’s parameters are optimized from data before deployment for prediction.
• fine-tuningAdapting a pretrained model to a new task by continuing training, usually with a smaller learning rate or on additional data.
• attentionA mechanism that assigns context-dependent weights to parts of a sequence so a model can emphasize some elements over others when processing input.
• token idInteger identifiers assigned to tokens so text can be represented as discrete numeric symbols.
• feature engineeringThe manual design or transformation of input variables into representations better suited for a learning algorithm.
• linear layerA linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.
• convolutional neural networkA neural network built from convolution and pooling layers that processes data with local spatial structure.
• linear regressionA transformation that preserves linear structure and can be represented by matrix multiplication.
• LSTMA long short-term memory recurrent neural network architecture designed to capture long-range dependencies in sequences.
• KerasThe programming interface of Keras that exposes multiple model-construction workflows and lets them interoperate.
• latent spaceAn encoded internal representation in latent space that preserves the structure needed for generation or manipulation.
• convolutional layerA two-dimensional convolution layer used here with a 1×1 kernel to project a residual to a matching shape.
• backpropagationA PyTorch tensor method that triggers backpropagation from a loss tensor and fills gradients on participating tensors.
• encoderThe sequence model component that reads the source sequence and compresses it into a representation for another model component to use.
• pretrainingThe initial large-scale training phase used to learn broad language representations before task-specific fine-tuning.
• objective functionA function that compares predictions with targets and returns a scalar indicating how wrong the model is for a given task.
• arrayA numerical container whose elements are organized over one or more axes and manipulated by vectorized operations.
• manifoldA lower-dimensional geometric subspace that is locally similar to Euclidean space, used here to describe the structure of valid data samples.
• regularizationA method that constrains a model during training to reduce overfitting and improve generalization.
• stateA hidden-state tensor carried by a recurrent network that summarizes previous inputs and influences future outputs.
• decodeThe part of an autoencoder or variational autoencoder that maps a latent representation back into an output space such as images.
• validation_data argumentA fit() argument that supplies a separate dataset for computing validation loss and metrics during training.
• neural network layerA neural-network building block that transforms one or more input tensors into one or more output tensors and may carry learned state.
• samplingThe process of drawing values or examples from a probability distribution, here to generate new content from a learned space.
• JAXA numerical computing and machine learning library for high-performance array computation, automatic differentiation, and compilation.
• PyTorchA Python machine-learning library used here as one of the frameworks whose code can run element-wise operations on GPUs; only PyTorch is actually a library entry in the glossary, but the section groups it with JAX and TensorFlow as frameworks.
• residual connectionA residual connection is a neural-network pathway that carries an earlier representation around one or more transformations to a later computation, where it can be combined with the transformed representation.
• tokenA discrete unit in a text sequence, such as a word or subword, that serves as the basic item attended over by the model.
• KerasHubA Keras-hosted library providing pretrained models and high-level tasks such as Stable Diffusion and TextToImage.
• tokenizationThe process of converting text into a sequence of discrete units such as words, characters, or punctuation marks.
• preprocessingThe transformation of raw data into a form suitable for input to a model.
• feature mapA spatial grid of activations produced by a convolutional or pooling layer, where each channel encodes the presence of a learned feature over locations.
• tf.data APITensorFlow's input पाइപ്പ്-line API for building efficient data preprocessing and loading pipelines.
• encoder-decoder architectureA model that maps an input sequence to an output sequence, often used for translation or generation tasks.
• subword tokenizationA tokenization method that splits text into units smaller than words but larger than characters to balance vocabulary size and sequence length.
• vocabularyThe complete set of tokens considered by a text representation scheme, defining the positions in the output vector.
• dropoutA neural-network regularization technique that randomly zeroes activation entries during training and behaves differently at inference time.
• max poolingA two-dimensional pooling layer that downsamples feature maps by taking local maxima.
• forward passThe phase of evaluating a computation graph from inputs to outputs to obtain activations and a final loss.
• interpolationA program-like behavior produced by a model that interpolates within trained examples rather than synthesizing novel algorithms.
• labelThe desired output values that a model is trained to match for each input example.
• GRUA recurrent neural network unit designed to process sequences with gated state updates.
• promptA short text description used to condition image generation, including both positive and negative forms in this section.
• digital imageData in which each example is a 2D image with one or more color channels, typically stored as a rank-3 tensor per sample.
• diffusion modelA diffusion-based generative model that starts from noise and iteratively denoises it into an image.
• evaluation datasetA held-out subset of the training data used during training to monitor performance and tune choices without touching the final test set.
• TensorFlowA Python-based open source machine learning platform with an ecosystem of components for training, deployment, and related workflows.
• machine learningA class of methods that learn to map inputs to outputs by searching for useful representations and rules from example data and feedback.
• baselineA simple reference method or model used to judge whether a machine-learning system outperforms a naive or trivial alternative.
• stochastic gradient descentA gradient-descent training method that computes updates from small batches of training examples rather than the full dataset at once.
• gradient descentA parameter update step computed from gradients to reduce model error.
• data augmentationA regularization approach that creates additional training examples by applying label-preserving random transformations to existing samples.
• expressive powerThe capacity of a model architecture to represent more complex functions or patterns.
• batch sizeThe number of sequence examples processed together in one forward or training step.
• bidirectional rnnAn RNN architecture that processes a sequence in both forward and reverse directions and merges the resulting representations.
• training stepA single update of a model on one batch, consisting of prediction, loss computation, gradient computation, and weight adjustment.
• normalizationThe process of rescaling or normalizing features so they have more stable numerical properties for learning.
• manifold hypothesisThe assumption that high-dimensional data lie near a lower-dimensional manifold, making smooth interpolation useful for learning.
• variational autoencoderA generative latent-variable model that learns an encoder-decoder structure and a probabilistic latent space for reconstructing and sampling data.
• vanishing gradient problemA training problem in deep networks where gradient signals become too small to effectively update earlier layers.
• data parallelA distributed-training method where each device holds a full replica of the model and processes different sub-batches in parallel.
• model parallelismA distributed-training method where different parts of one model are placed on different devices to fit or execute a larger model.
• prefetchA data-input optimization that overlaps data preparation with model execution by preparing upcoming batches in advance.
• hyperparameterA user-chosen model or training setting, such as layer count or activation choice, that is set before training and tuned externally rather than learned by backpropagation.
• quantizationThe technique of reducing numeric precision of model weights or activations to speed up computation while preserving output behavior as much as possible.
• distributed trainingA training approach that spreads computation and/or data across multiple devices to speed up model training.
• hyperparameter tuningThe process of searching over model configuration choices to find settings that improve validation or test performance.
• convolution kernelThe learned weight tensor used by a convolution to transform local input patches.
• TensorFlow DatasetTensorFlow's iterable dataset abstraction for efficient loading, preprocessing, and batching of data.
• Vectorization layerA Keras preprocessing layer that tokenizes text and converts it into encoded vectors such as integer sequences or multi-hot representations.
• keyA JAX random-state value passed explicitly to PRNG operations to generate deterministic random numbers without global state.
• multi-head attentionAn attention variant that runs several attention computations in parallel with different parameters to learn different relationships.
• positional embeddingA representation added to sequence elements to preserve order information for models that otherwise treat inputs as a set.
• gradientThe generalization of a derivative to a tensor-valued function, representing the local rate and direction of change with respect to each input component.
• multi-hot encodingA representation scheme where multiple positions are set to 1 and all others to 0 to indicate the presence or absence of several categories or symbols.
• computation graphA directed acyclic graph that represents a computation as operations and intermediate values, enabling automatic differentiation.
• tf.GradientTapeTensorFlow’s context object for recording tensor operations so gradients can later be computed from the recorded graph.
• automatic differentiationA mechanism that programmatically computes exact derivatives of recorded operations by applying the chain rule to a computation graph.
• dataset splitOne of the K disjoint subsets created when data is split for cross-validation.
• data transformationA transformation that maps data or representations within or between geometric spaces.
• multilabel classificationA classification task in which each example may belong to multiple classes at the same time, even though there are multiple possible classes overall.
• model architectureThe structural design of a deep learning model, here contrasted with approaches that adapt at test time rather than only scaling fixed architectures.
• production dataReal operational data encountered after deployment, used here as the reference for estimating future performance.
• classification headThe final neural-network layer that maps learned representations to class predictions.
• productionA production environment is the live operational setting in which software or other systems serve real users, workloads, or business processes.
• weight decayWeight decay is a model-training regularization technique that discourages large parameter values by penalizing their magnitudes or shrinking them during optimization.
• Functional APIThe Keras model-building interface that expresses a model as a directed acyclic graph of layers with explicit connectivity.
• patchA local subregion of an image that is extracted and treated as one sequence element in ViT processing.
• optimizationThe procedure that updates model parameters based on the loss, typically by a variant of stochastic gradient descent.
• parameterA model parameter that is updated by backpropagation during training.
• stateless systemA programming design in which functions do not depend on or modify hidden global state and produce outputs only from explicit inputs.
• data annotationThe process of having people manually label or correct dataset examples.
• batch normalizationA normalization technique used in neural networks that stabilizes activations during training, commonly implemented as a normalization layer.
• cross-validationA validation procedure that partitions data into K folds, trains on K−1 folds, and evaluates on the remaining fold in rotation.
• machine learning workflowThe end-to-end sequence of steps for defining, building, evaluating, deploying, and maintaining a machine learning system.
• program synthesisA technique that automatically constructs programs satisfying a specification by searching over candidate code.
• torch.nn.ModuleThe base class for defining stateful PyTorch models with parameters and a forward method.
• SimpleRNN layerA Keras recurrent layer that processes sequences by iterating a simple recurrent cell over timesteps.
• tokenizerA component that converts text into a sequence of integer token IDs using a vocabulary and tokenization rules.
• bounding boxA rectangle representation used to localize an object in an image, typically encoded by coordinates and size.
• concept driftA change over time in the statistical properties of production data that can degrade a deployed model's performance.
• pipelineA supported way of using Keras that combines model-building and training choices for a particular level of flexibility.
• KerasTunerA Keras-related library for automating hyperparameter search over model configurations.
• standardThe preprocessing step that normalizes text before tokenization, such as lowercasing.
• channelA component of a layer output corresponding to one class or feature channel, used here to select the top predicted class.
• activationThe numeric value of a model activation.
• TPUA specialized tensor-processing chip and cloud-accessible hardware accelerator used to speed up machine learning training.
• cross entropy lossA classification loss that measures the mismatch between predicted probability distributions and true token labels.
• translation invarianceThe property that a learned pattern can be recognized at different spatial locations without relearning it from scratch.
• TensorFlow ServingA serving library for deploying TensorFlow computation graphs with saved weights as inference services.
• TFLiteA framework for efficient on-device TensorFlow inference on mobile and embedded hardware.
• from_preset()A constructor used to load a pretrained model’s weights, configuration, and tokenizer assets from a preset.
• jax.value_and_gradA JAX function that returns both a function value and its gradient in one call.
• SAMA pretrained segmentation model available through KerasHub for prompting-based segmentation tasks.
• early stoppingA training strategy that stops optimization when validation performance stops improving.
• search spaceThe set of all hyperparameter choices that a tuning process is allowed to consider for a model.
• sequence modelA model designed to ingest an ordered token sequence and learn dependencies across positions in that sequence.
• image segmentationA vision task that assigns a label to each pixel or region of an image, providing denser localization than object detection.
• ONNXAn open format and runtime ecosystem for representing and executing trained machine-learning models outside the original training framework.
• pruningA model-compression technique that removes less important weights or connections to reduce model size and cost.
• RESTREST is an architectural style for distributed hypermedia systems that organizes interactions around resources and representations through constraints including a uniform interface, stateless communication, cacheability, and layered components.
• TensorFlow.jsA JavaScript TensorFlow ecosystem library for running models in browsers or other JavaScript environments.
• depthwise separable convolutionA convolution variant that applies per-channel spatial convolution followed by pointwise channel mixing to reduce parameters and computation.
• confidenceThe model's estimated certainty for a predicted detection, often represented as a score or probability.
• ModelCheckpointA Keras callback that saves a model or its best weights during training at specified points or when performance improves.
• YOLOA family of single-stage object-detection models that predict bounding boxes and classes directly from an image in one forward pass.
• feedforward networkThe Transformer submodule that applies a position-wise dense transformation and nonlinearity to each token representation.
• generative modelingAI systems that generate new text, images, or other content in response to prompts by modeling the distribution of data.
• bag-of-words modelA text model that ignores token order and represents a document by which words occur in it, often with per-word weights.
• data collectionThe process of gathering data examples for a machine learning project before model development.
• calibration dataData whose distribution matches the real-world inputs a model will encounter after deployment.
• selection biasA bias introduced when the data collection process produces a sample that is not representative of the target population.
• gridA grid is a regular arrangement of positions or cells organized by rows and columns or by fixed neighboring relationships.
• variableA mutable TensorFlow value that stores learnable parameters such as weights and bias in this example.
• artificial intelligenceComputer systems that perform tasks associated with human intelligence, used here in a broad modern sense that the text argues is often closer to automation.
• prompt engineeringThe practice of crafting prompts to elicit better performance from a large language model on a task.
• adamAn adaptive gradient-based optimization algorithm that updates model parameters using running estimates of first and second moments of gradients.
• mean absolute errorThe average absolute difference between predictions and numeric targets, used as a regression error metric.
• reinforcement learningA learning setup where an agent improves by interacting with an environment and receiving feedback, used here as a way to keep generating fresh training problems for LLMs.
• natural language processingThe field of using computers to interpret, analyze, or generate human language, especially written text.
• ReLUA rectified linear activation operation applied element-wise to tensor entries, returning the input when positive and zero otherwise.
• machine learning problemA task framed so that a model can be trained on data to make useful predictions or decisions.
• inner productAn inner-product operation used as a similarity score between vectors.
• hierarchyA nested collection structure of tuples, dicts, and lists used by JAX to represent composite inputs and states.
• hybrid approachA model that combines deep-learning components with algorithmic or symbolic components.
• bounding box predictionThe model output that estimates object-localizing rectangle coordinates for an image region.
• modularity-hierarchy-reuse formulaA design principle that structures a system into modules, arranges them hierarchically, and reuses the same modules in multiple places.
• ensembleA technique that combines multiple models or views to improve performance by exploiting complementary errors or representations.
• GPT modelA decoder-only transformer language model that generates text by predicting the next token from left-context only.
• MNIST datasetA standard dataset of handwritten-digit images used for machine-learning experiments and examples.
• single-stage detectorAn object-detection architecture that predicts bounding boxes and class labels in one pass with a single model, rather than using separate proposal and classification stages.
• segmentation maskAn image-shaped target annotation for segmentation where each pixel stores the class or region of the corresponding input pixel.
• greedy algorithmA decoding strategy that selects the most likely predicted token at each step without considering future alternatives.
• abstractionAn abstraction level that exposes simpler interfaces while hiding low-level implementation details.
• top-k samplingA decoding method that limits token selection to the K most probable candidates before sampling.
• validation lossThe loss value computed on validation data to estimate how well a model generalizes beyond the training set.
• device meshA grid-like arrangement of devices used to assign different parallelism roles along named axes.
• LayoutMapA Keras API for assigning variable sharding or replication layouts across a device mesh.
• replicationThe practice of copying a tensor or variable so multiple devices hold the same value.
• maskA technique for marking padded or irrelevant sequence positions so later layers ignore them.
• text augmentationA family of methods that create modified text examples to improve model robustness.
• jax.gradA JAX function that creates a gradient-computing function for differentiable inputs and outputs.
• cognitive automationThe use of AI to automate tasks by encoding and operationalizing human skills and knowledge rather than achieving full human-like intelligence.
• local patternA pattern that occurs within a limited neighborhood of the input rather than across the whole input space.
• paddingPadding is the addition of extra space, values, or data around or within a structure to satisfy size, alignment, layout, or processing requirements.
• tensorA tensor with three axes, used here for an image or feature-map representation with height, width, and depth.
• pseudo-random number generatorA pseudorandom number generator is a deterministic computational process that uses an initial seed or internal state to produce a sequence with random-like statistical properties.
• derivativeThe mathematical process of computing derivatives, used here to support gradient-based learning.
• training datasetThe subset of data used to fit a model's parameters from examples, here emphasized as being small in size.
• autoencoderA neural network trained to reconstruct its input, often by compressing and then decoding it, used here for denoising images.
• intelligenceThe capacity to understand, learn, and adapt to tasks and environments, discussed here as the broad property AI aims to approximate.
• mutationUpdating an existing tensor or array object directly rather than constructing a new one.
• iterated K-fold validation with shufflingA repeated cross-validation protocol that cycles through multiple train/validation splits to obtain a more reliable estimate.
• jax.jitA JAX decorator that compiles a function with XLA for faster execution.
• jax.random.splitA JAX function that deterministically derives new random keys from an existing key.
• IMDb datasetA movie-review dataset used here as the source of training and validation reviews for fine-tuning.
• source sequenceA sequence of source-token vectors that attention scores and combines to support prediction of a target token.
• target sequenceThe sequence of target-token vectors being decoded, whose current element queries the source sequence in attention.
• einsum notationA notation for expressing tensor operations by labeling axes with letters and specifying multiplication and summation by repeated indices.
• freezeTo keep selected model parameters unchanged during training so they do not receive optimizer updates.
• single-label categorical classificationA classification task in which each example belongs to exactly one class from a set of mutually exclusive classes.
• denoisingDenoising is the process of removing unwanted noise or corruption from data while preserving meaningful structure.
• indexingThe stage of text preprocessing that maps tokens to integer indices for numeric model input.
• shuffleTo randomly reorder data before splitting it so that contiguous ordering does not bias the partition.
• mixed precisionA training technique that uses lower-precision numeric types for some computations while keeping others in higher precision to balance speed, memory use, and numerical stability.
• sliding windowA technique that moves a fixed-size window across text or other data to create overlapping local segments.
• Abstraction & Reasoning Corpus for Artificial General IntelligenceA benchmark dataset designed to measure generalization and abstract reasoning on tasks that are easy for humans but challenging for standard AI systems.
• chain ruleThe differentiation rule for composing functions, used to compute gradients through layered neural-network operations.
• class activation heatmapA visualization technique that highlights image regions relevant to a specific predicted class by producing class-specific activation heatmaps.
• GPUA graphics processing unit used here as accelerated hardware for parallel computation in deep learning workloads.
• language modelingA next-token prediction training setup in which the model learns to predict the next token from previous tokens.
• carry stateThe LSTM internal state that transports information across timesteps to preserve long-term context.
• spatial hierarchyA layered organization of features in which deeper layers represent patterns over increasingly large spatial extents of the input.
• scalabilityThe practice of increasing model size, data, or training compute to improve performance in machine learning.
• interpretabilityThe degree to which a model's behavior can be understood and interpreted by humans.
• preferenceThe relative ranking of candidate responses by people according to what is more helpful or desirable.
• reward modelA model that scores candidate outputs according to how well they match human preferences.
• reinforcement learning with human feedbackA training setup that uses human preference feedback to build a reward signal for improving a model’s outputs.
• reward signalFeedback value used in reinforcement learning to indicate how desirable an action or output is.
• tensor operationAn elementary array/tensor computation such as matrix multiplication, rectified linear activation, or normalization used as a building block in neural networks.
• backendThe underlying execution framework that a machine-learning library can target to run models.
• recurrent dropoutA dropout variant for recurrent networks that applies a temporally consistent mask to recurrent computations.
• tf.functionA TensorFlow decorator that compiles a Python function into a faster executable form, typically graph mode.
• byte pair encodingA lossless-compression algorithm adapted here to build subword vocabularies by merging frequent symbol pairs.
• InputA Keras constructor used to create symbolic input objects that specify the expected shape and dtype of model inputs.
• sparse_categorical_crossentropyA classification loss function for integer-coded class targets that measures mismatch between predictions and true labels.
• ConvNet architecture principleA general design guideline for building convolutional neural networks, including block structure, filter progression, and regularization choices.
• next-token predictionA language-model training objective that teaches the model to predict the next token in a sequence from previous tokens.
• image_dataset_from_directoryA Keras utility function that builds a dataset from image files organized in class-specific subdirectories.
• build()A Keras model method that creates weights for a model when given an input shape before the model is first called.
• GemmaA pretrained large language model family used here for text generation and later instruction fine-tuning.
• visual encoderA model component that converts images into vector representations suitable for a larger multimodal model.
• code compilationThe process of converting code into a compiled form for execution, used here as a performance and debugging tradeoff in TensorFlow, JAX, and PyTorch.
• categorical encodingA representation scheme where one position is 1 and all others are 0 to encode a discrete category or symbol.
• regular expressionA pattern language for matching and manipulating character sequences in text.
• out-of-vocabulary wordA special unknown-token marker used when a tokenizer cannot represent a word with known vocabulary items.
• Sequential modelA Keras class for building models as a linear stack of layers.
• .gradA PyTorch tensor attribute where accumulated gradients are stored for tensors that track gradients.
• subclassed layerA Keras workflow pattern where models or layers are defined by writing Python classes or functions directly instead of assembling prebuilt layers with Sequential or the Functional API.
• categorical dataData whose values belong to discrete categories rather than a continuous numeric scale.
• causal language modelA language model trained to predict the next token from past tokens only, without access to future context.
• cachingCaching is the practice of storing reusable data or computation results so later access can avoid repeating retrieval or computation.
• matrix-matrix multiplicationThe linear-algebra operation that multiplies matrices or higher-rank tensors in a way that composes transformations.
• sigmoidA statistical classification method that models class probabilities with a linear predictor passed through a logistic function.
• PyTorch backendThe execution backend that allows Keras workflows to run on top of PyTorch.
• data modalityA type of data such as text, images, or audio that can be represented as sequences for model processing.
• XLAAccelerated Linear Algebra, TensorFlow's optional compiler used to further optimize and speed up function execution.
• videoData in which each example is a sequence of image frames, typically stored as a rank-5 tensor for a batch.
• moduleA repeated architectural unit in the Transformer made from attention and feedforward sublayers.
• queryAn attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.
• valueIn attention, a value is a vector representation of input content that is combined with other values according to attention weights to produce an output.
• symbolic AIAn AI approach that represents knowledge and reasoning using explicit symbols and rules rather than learned representations.
• L1 regularizationA regularization method that penalizes the absolute values of weights to encourage sparsity.
• graphA node-and-edge structure used here to represent how model layers are connected, including branches and merges.
• symbolic tensorA tensor-like placeholder that encodes shape and dtype information without holding actual data values.
• deep learning frameworkSoftware libraries for building and training neural network models.
• beam searchA decoding method that keeps a fixed number of candidate generation paths and expands them step by step.
• unsupervised taskA learning task that does not require labeled targets and instead derives supervision from the data itself.
• GlobalAveragePooling2DA pooling layer that averages each feature map across its spatial dimensions to produce one value per channel.
• architecture priorsAn inductive bias about which model structures are likely to work well for a task before training on data.
• call()A Keras layer or model method that defines the forward computation applied to inputs.
• noiseAn initial image-like input made of unstructured random values rather than meaningful visual content.
• callbackA user-defined callback that runs automatically during training to perform extra actions such as monitoring outputs.
• vectorizationThe process of converting data into vector form so it can be used by machine-learning code.
• chain-of-thought promptingA search-based inference approach that explores reasoning in natural language by generating intermediate thought steps before an answer.
• loss scalingA technique that multiplies the loss by a scalar factor to increase gradient magnitudes and reduce underflow in low-precision training.
• image generation modelA model that generates images, here assumed to operate at large scale and potentially use ViT-style architectures.
• datasetCollections of examples used to train or evaluate machine learning models, here including large-scale internet-derived media and text corpora.
• diversityIn ensemble learning, the degree to which models make different errors or rely on different evidence.
• hold-out validationAn evaluation protocol that splits data once into separate subsets for training, validation, and/or testing.
• data leakageThe unintended transfer of information from evaluation data into model choices, causing biased estimates of generalization.
• missing valueAn absent data field in a sample that must be handled explicitly during preprocessing.
• activation functionA nonlinear function applied to a layer’s output to reshape it and increase model expressiveness.
• feature pyramid networkAn architecture that merges multi-scale feature maps from different depths to support detection of objects at multiple sizes.
• sequenceAn ordered sequence of integer values used here to represent tokenized text.
• coordinate changeA transformation that converts data points into a different coordinate system to obtain a more useful representation.
• Order propertyInformation carried by the order of tokens in text, as opposed to only their presence or count.
• layer normalizationLayer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
• sequence generationThe task of producing an ordered sequence of tokens, characters, or other elements from a model rather than a single label.
• adversarial inputInputs intentionally or effectively perturbed to cause a model to make a wrong prediction despite minimal change to the original example.
• performanceA measurement of business outcome or value used to judge the effect of a deployed model.
• reconstruction lossA loss that measures how well a model reproduces its input or a corrupted version of it.
• RGBA color representation that encodes images using red, green, and blue channels.
• weighted averageAn averaging operation that assigns different weights to different values before combining them into a single result.
• BERTA pretrained bidirectional Transformer encoder model for language understanding, used here as an NLP example rather than a generation model.
• data representativenessThe degree to which a validation set reflects the distribution and variety of the data the model will face in practice.
• prompt encoderA model component that converts a prompt into an embedded vector representation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.