liveBookmanning.com

What you will learn in Deep Learning with R, Third Edition

From first principles to generative AI

by François Chollet, Tomasz Kalinowski

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• language modelA large language model is a machine-learning model trained on large text corpora to understand and generate natural-language text.
• generalizationGeneralization is a model's ability to perform accurately on previously unseen inputs from the same underlying domain as its training data.
• attentionAn attention layer computes context-dependent representations by allowing sequence elements to weight information from other elements.
• feature engineeringFeature engineering is the manual design or transformation of input features into representations suitable for machine learning.
• latent spaceA latent space is a vector representation space in which tokens are positioned so that geometric relationships encode learned semantic or contextual relationships.
• recurrent neural networkA recurrent neural network is a neural-network model that processes sequences recurrently by carrying information through an internal state across positions.
• transformer architectureA Transformer is a neural-network architecture designed to process sets or sequences using attention-based contextual representations.
• overfittingOverfitting is a modeling condition in which performance is better on training data than on previously unseen data, including when repeated tuning adapts a model to validation data.
• feature mapA feature map is a spatial array of activations indicating the presence or strength of a learned visual pattern at different image locations.
• foundational modelA foundation model is a large pretrained model intended to support many downstream tasks with little or no task-specific retraining.
• JAXJAX is a Python library for high-performance numerical computing and machine learning with composable transformations such as automatic differentiation and just-in-time compilation.
• fine-tuningFine-tuning is the continued training of a pretrained model, typically with a small learning rate, to adapt its learned representations to a new task.
• embeddingAn embedding is a learned numerical vector representation of an object, token, or other data item.
• gradientA gradient is the tensor-valued derivative of a function whose inputs are tensors, with each coefficient indicating how the output changes with respect to one input coefficient.
• preprocessingData preprocessing is the preparation of raw data by reshaping, scaling, encoding, or otherwise converting it into a model-compatible form.
• regularizationRegularization is a modeling technique that constrains learning to reduce overfitting and improve generalization to unseen data.
• vocabularyA vocabulary is the finite set of tokens that a tokenizer recognizes and assigns indices for text representation.
• tensorA JAX array is JAX's numerical array object representing values on which transformed and accelerated computations can operate.
• KerasKeras is an open-source deep learning API for building and training neural networks.
• tokenizerA tokenizer is a text-processing component that converts input text into a sequence of discrete tokens, often assigning each token an integer index.
• expressive powerRepresentational power is the ability of a model to encode and express distinctions or patterns in data.
• neural network layerThe Keras Layer class is a base API class for defining reusable neural-network computation components.
• objective functionA loss function is a function that assigns a numerical measure of discrepancy between a model's prediction and the true target.
• convolutional neural networkA convolutional neural network is a neural network that applies convolution-based operations to learn spatial or locally structured features, especially from grid-like data such as images.
• pretrainingUnsupervised pretraining is training a model on input data without manually supplied labels before adapting it to downstream tasks.
• tensor operationA tensor operation is a mathematical or computational transformation that takes one or more tensors as input and produces tensor values.
• training datasetTraining data is the collection of examples from which a machine-learning model learns its parameters or representations.
• PyTorchPyTorch is an open-source machine-learning library providing tensor computation and automatic differentiation capabilities.
• classifierA classification problem requires assigning inputs to discrete categories based on their characteristics.
• LSTMLong short-term memory is a gated recurrent-neural-network architecture designed to preserve and control information across sequence timesteps.
• generationThe computational generation of natural-language text by a model.
• machine learningMachine learning is an approach in which computer systems improve their performance on tasks by learning from data or experience rather than relying solely on explicitly programmed rules.
• gradient descentGradient descent is an iterative optimization algorithm that updates parameters in the direction opposite to a function's gradient to reduce a loss or objective.
• tokenA token is a discrete unit of text processed by a language model, such as a word, subword, character, or punctuation symbol.
• linear layerA Dense layer is a neural-network layer in which every output unit is connected to every input unit through learned weights and biases.
• data augmentationData augmentation is a technique that generates additional training examples by applying label-preserving transformations to existing data.
• convolutional layerA convolutional layer is a neural-network layer that applies learned local filters across spatial or sequential input dimensions.
• TensorFlowTensorFlow is an open-source platform and software ecosystem for numerical computation and machine learning.
• KerasHubKerasHub is a library that provides reusable components and utilities for working with pretrained and generative machine-learning models in Keras.
• linear regressionA linear transform is a mapping that preserves vector addition and scalar multiplication and can be represented by matrix multiplication.
• diffusion modelA diffusion model is a generative model that iteratively transforms noise into structured data by reversing a process that progressively adds noise.
• evaluation datasetValidation data is a reserved subset of examples used to assess a model during or after training without updating its parameters.
• tokenizationWord tokenization is the division of text into tokens corresponding primarily to words.
• backpropagationBackpropagation is an algorithm for computing gradients of a neural network's loss with respect to its parameters by applying the chain rule backward through the network.
• training stepA training step is one iteration that evaluates a model on a batch, computes its loss and gradients, and updates its parameters.
• network topologyNetwork topology is the structural arrangement and connectivity pattern of a neural network's layers and operations.
• parameterA weight is a learned numerical parameter that determines part of a neural-network transformation, including an embedding lookup.
• hyperparameterA hyperparameter is a configuration value selected outside parameter learning that controls a model, training procedure, or optimization process.
• interpolationInterpolation is the estimation of values between observed data points under an assumed continuity or smoothness relationship.
• dropoutDropout is a regularization technique that randomly disables units during training to reduce reliance on particular activations.
• baselineA baseline is a reference level of performance produced by a simple strategy or comparison group against which a model's performance is evaluated.
• subword tokenizationA subword token is a token representing a complete word, a word fragment, or a character-level fallback unit.
• local generalizationLocal generalization is the ability to make useful predictions for new inputs that remain similar to previously observed examples.
• abstractionAn abstraction is a compact, reusable representation of underlying structures, principles, or invariants extracted from experience or data.
• arrayAn R array is R's built-in multidimensional array data structure for storing values across one or more dimensions.
• architecture priorsAn architecture prior is an inductive structural constraint built into a model architecture that favors particular forms of representations or computations.
• optimizationOptimization is the task of finding input or parameter values that minimize or maximize an objective function.
• manifoldA manifold is a continuous surface or space that locally resembles a simpler coordinate space and can represent the structure of data.
• encoderAn encoder model transforms an input sequence into an intermediate representation that summarizes information from that sequence.
• decodeA decoder network is a neural-network component that maps a latent representation back to an output such as a reconstructed image.
• segmentation maskA segmentation mask is a spatial label representation identifying the pixels belonging to an object or region in an image.
• hypothesis spaceA hypothesis space is a predefined set of candidate operations, representations, or rules that a learning algorithm searches.
• sequenceTime series data consists of observations ordered by time, often recorded at regular intervals and exhibiting temporal dynamics such as trends, cycles, regimes, and spikes.
• training loopA training loop is a repeated process that evaluates examples, adjusts model parameters, and progressively minimizes the loss function.
• max poolingA max-pooling layer down-samples local regions by retaining their maximum values.
• accuracyAccuracy is the proportion of evaluated predictions that exactly match their corresponding target labels.
• artificial intelligenceArtificial intelligence is the field of building systems that perform tasks associated with human intelligence, such as reasoning, perception, language use, and decision-making.
• automatic differentiationAutomatic differentiation computes exact derivatives of programs composed of differentiable operations by systematically applying the chain rule.
• performanceA metric is a numerical measure of model performance monitored during training or validation without necessarily being optimized directly.
• reinforcement learningReinforcement learning is a machine-learning technique in which an agent learns behavior through interaction with an environment and feedback such as rewards.
• jax.value_and_gradjax.value_and_grad is a JAX transformation that constructs a function returning both a function's value and its gradient.
• tf.data APIThe tf.data API is TensorFlow's API for constructing input pipelines that process and preprocess data efficiently.
• tf.GradientTapetf.GradientTape is a TensorFlow context-based API object that records tensor operations for subsequent gradient computation.
• prior knowledgePrior knowledge is information or structural assumptions available to an agent before it learns from or encounters the current problem.
• dataset splitSharding is the partitioning of a variable or data object into distinct chunks assigned to different devices.
• labelGround truth is the set of target annotations for a dataset, typically collected from human judgments or measurements.
• code compilationPyTorch compilation is the transformation of PyTorch computations into an optimized executable representation.
• embedding layerAn embedding layer maps discrete integer indices to learned dense vector representations.
• generative modelingGenerative AI comprises computational models that produce new content such as text, images, audio, or code from learned patterns.
• program synthesisProgram synthesis is the automatic generation of programs that satisfy specified input-output behavior or other formal requirements.
• stateless systemStatelessness is a design property in which operations do not depend on hidden mutable state outside their explicit inputs and outputs.
• freezeA frozen parameter is a model parameter excluded from optimization so that its value remains unchanged during further training.
• intelligenceIntelligence is the ability to use available information and past experience to produce successful behavior in uncertain and changing situations.
• promptA prompt is an input specification that identifies the object or region for which an image-segmentation model should produce a mask.
• batchingA batch, or mini-batch, is a subset of training examples processed together in one model evaluation and parameter update.
• vanishing gradient problemA vanishing gradient is a training condition in which gradients become extremely small as they propagate through a deep network, slowing or preventing parameter updates in earlier layers.
• hyperparameter tuningHyperparameter tuning is the process of selecting model configuration values by evaluating alternative settings during development.
• data parallelData parallelism is a distributed-training technique that replicates a model across devices and processes different data subsets on those devices.
• model parallelismModel parallelism is a distributed-training strategy that partitions a model across devices so different devices execute different model components.
• variational autoencoderA variational autoencoder is a probabilistic autoencoder whose encoder represents inputs as parameters of a latent probability distribution and whose decoder reconstructs or generates data from sampled latent points.
• multiclass classificationA multiclass classification model is a predictive model that assigns inputs to one of more than two classes and produces scores or probabilities for those classes.
• pipelineA workflow is an organized sequence of modeling or training operations that provides a particular balance of usability and flexibility.
• quantizationModel quantization is the conversion of model computations or parameters to lower-precision numerical representations to reduce resource requirements.
• probability distributionA data distribution is the probability distribution governing the inputs, examples, or outcomes in a machine-learning task.
• GRUA gated recurrent unit (GRU) is a recurrent neural-network architecture that models sequence dependencies with update and reset mechanisms.
• distributed trainingDistributed training is the execution of model training across multiple processors or machines that coordinate computation and parameter updates.
• computation graphA computation graph is a directed representation of operations and their data dependencies used to evaluate computations and derivatives.
• YOLOYou Only Look Once is a family of single-stage object detection models that predict object locations and categories in a single network evaluation.
• depthwise separable convolutionA depthwise separable convolution factorizes convolution into independent spatial convolutions over input channels followed by a pointwise convolution that mixes channels.
• Vectorization layerTextVectorization is a Keras preprocessing layer that transforms text into numerical representations using a learned vocabulary and configurable tokenization.
• GPUA GPU is a massively parallel processor designed primarily for graphics computation and also widely used for numerical machine learning workloads.
• image segmentationImage segmentation is the task of assigning semantic labels to individual pixels or regions of an image.
• convolution kernelA convolution kernel is a learned weight matrix that transforms each extracted input patch into an output feature vector.
• single-stage detectorA single-stage object detector predicts object locations and categories directly in one integrated model pass rather than first generating region proposals.
• Functional APIThe Keras Functional API is an interface for constructing models by connecting layer operations as a directed computation graph.
• feedforward networkA feedforward block is a neural-network subnetwork that expands a representation, applies a nonlinearity, and projects it back to the model dimension.
• batch normalizationBatch normalization is a neural-network normalization technique that standardizes layer activations using mini-batch statistics during training and stored moving statistics during inference.
• patchAn image patch is a localized rectangular portion of an image processed as an individual visual input unit.
• learning rateA learning rate is the step-size hyperparameter controlling how far model parameters move during an optimization update.
• pretrained modelA pretrained model is a machine-learning model whose parameters have already been learned from prior training data.
• ReLUA rectified linear unit is the activation function that maps an input x to max(x, 0).
• forward passA forward pass is the propagation of input values through a computational graph in the direction of its outputs.
• stochastic gradient descentMini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients computed from randomly sampled small batches of training data.
• prompt engineeringPrompt engineering is the systematic design and refinement of inputs to a generative model to improve its performance on a task.
• algorithmic moduleAn algorithmic module is a composable computational component that performs discrete operations such as formal reasoning, search, or abstraction.
• chain ruleThe chain rule is a calculus rule for differentiating a composition of functions by multiplying derivatives of the component functions.
• data curationData curation is the selection, cleaning, organization, and quality management of data for analysis or machine learning.
• latent manifoldA latent manifold is a low-dimensional, structured subspace in which encoded data points can be continuously interpolated.
• manifold hypothesisThe manifold hypothesis is the claim that natural data lies on a low-dimensional manifold within the high-dimensional space in which it is embedded.
• eager evaluationEager execution is an execution mode in which operations run immediately and produce results without first constructing a deferred computation graph.
• data annotationData annotation is the process of assigning human-provided labels or other metadata to data examples.
• weight decayL2 regularization is a regularization method that adds a penalty proportional to the squared values of model parameters.
• fluid intelligenceFluid intelligence is the ability to solve novel problems and adapt reasoning across varied situations without relying solely on previously learned knowledge.
• cross-validationK-fold cross-validation is an evaluation procedure that partitions data into K subsets and repeatedly trains on K−1 subsets while evaluating on the remaining subset.
• scalabilityScalability is the ability of a system or method to handle increasing amounts of data or computation without becoming impractical.
• production dataProduction data is data generated by the real operating environment in which a deployed machine-learning model is used.
• sequence modelA sequence model is a model that processes an ordered sequence of tokens or other values while learning dependencies among their positions.
• GemmaGemma is a family of pretrained generative language models released in specialized instruction-tuned variants.
• data representationA data representation is a particular encoding or view of data that can make a task easier to solve.
• geometric moduleA geometric module is a composable computational component that represents or transforms data through continuous numerical operations and pattern recognition.
• multi-head attentionMultiHeadAttention is a Keras layer that computes attention through multiple parallel attention heads and combines their outputs.
• tensor shapeAn output shape is the ordered set of dimension sizes describing the tensor produced by a layer.
• dataset_mapdataset_map is a dataset transformation method that applies a callable transformation to each dataset element.
• prefetchData prefetching prepares future input elements asynchronously while current elements are being processed.
• KerasTunerKerasTuner is a library for defining, running, and evaluating hyperparameter searches for Keras models.
• deploymentDeployment is the process of making a trained machine-learning model available in an operational environment for use.
• residual connectionA residual connection is a neural-network connection that adds or otherwise passes an earlier representation to a later layer.
• artificial general intelligenceGeneral intelligence is broad intellectual capability that enables a system to perform a wide range of cognitive tasks rather than a narrowly defined task.
• tensor assignmentTensor assignment is the operation of storing values into a tensor, including modification of existing tensor elements.
• text-to-image modelA text-to-image model is a generative model that produces images corresponding to textual descriptions.
• paddingPadding is the addition of rows and columns around an input feature map so convolution windows can cover boundary locations.
• attention_maskA padding mask is a tensor of indicators identifying padded positions that should be ignored during sequence computation or loss evaluation.
• keyA key vector is an attention representation used to compute compatibility with a query.
• test-time adaptationTest-time adaptation is a method in which an AI system performs problem-specific learning or reasoning during evaluation using information from the test task.
• token idA token ID is a discrete integer identifier that selects a token representation from a vocabulary or embedding matrix.
• return_sequencesreturn_sequences is a Keras recurrent-layer configuration option that makes the layer output its full sequence rather than only its final-timestep output.
• batch dimensionA batch axis is the tensor axis indexing the samples grouped into a batch.
• coordinate changeA coordinate change is a transformation that expresses points using a different coordinate system.
• class probability mapA class probability map is a spatial output containing class probabilities for each grid location over the possible object classes.
• gradient-free optimizationGradient-free optimization is optimization that selects parameter values without using derivatives of the objective with respect to those values.
• search spaceA search space is the set of allowed values or choices for the hyperparameters being optimized.
• tunerA tuner is a hyperparameter-search component that selects configurations, builds and trains models, and records their metrics.
• temperatureTemperature is a generation parameter that controls the concentration or spread of a model's next-token probability distribution.
• standardStandardization is the normalization of text through transformations such as case conversion, punctuation removal, or encoding normalization.
• softmaxA softmax operation converts a vector of scores into nonnegative values that sum to one, typically representing class probabilities.
• program-centric abstractionProgram-centric abstraction is abstraction based on discrete executable structures that support reasoning, planning, and causal understanding.
• hold-out validationHold-out validation is an evaluation procedure that reserves a subset of available data for assessment without using it to fit the model.
• encoder-decoder architectureA sequence-to-sequence model predicts a target sequence from a separate source sequence, typically conditioning each target token on prior target tokens.
• hallucinationThe hallucination problem is a generative-model failure mode in which a model produces plausible-sounding but unsupported or incorrect content.
• ModelCheckpointModelCheckpoint is a Keras callback that saves model states during training, optionally retaining only the state with the best monitored performance.
• reconstruction lossA reconstruction loss measures the discrepancy between original data or representations and their model-generated reconstructions.
• from_preset()from_preset() is a KerasHub constructor that loads a pretrained model together with its configuration, weights, and tokenizer assets.
• validation lossValidation loss is the loss value computed from a model's predictions on validation data.
• iterated K-fold validation with shufflingIterated K-fold validation is an evaluation procedure that repeats K-fold validation across multiple partitions or repetitions to obtain a more accurate performance estimate.
• noiseNoise is a random component of an image representation that obscures its meaningful signal.
• data collectionDataset collection is the process of obtaining data instances for use in analysis or machine learning.
• call()A call method defines the forward computation performed when a neural-network layer or model receives inputs.
• strided convolutionA strided convolution is a convolution that moves its kernel by more than one input location at a time to reduce spatial resolution.
• parameter countModel size is the amount of structural and parameter capacity in a machine learning model, determined by features such as layer and unit counts.
• derivativeA derivative is a measure of how a function's output changes in response to an infinitesimal change in an input.
• trainingTraining is the process of adjusting a machine-learning model's parameters using data and an optimization procedure.
• fully connected neural networkA dense neural network is a neural network architecture in which units in one layer are connected to all units in the adjacent layer.
• mini-gptmini-GPT is a small generative language model implemented as a book-specific example.
• padding tokenA padding token is a placeholder token inserted to make sequences conform to a common length and carries no sequence content.
• out-of-vocabulary wordAn unknown token is a reserved token representing text units that are absent from a tokenizer's retained vocabulary.
• natural language processingNatural language processing is the computational analysis and generation of human language.
• geometric transformationA geometric transformation is a mapping that changes the position, scale, orientation, or representation of geometric objects.
• model optimizationModel optimization is the modification of a trained model to reduce computational, memory, or latency costs while preserving acceptable predictive performance.
• ONNXONNX is an open format and ecosystem for representing and exchanging machine-learning models across frameworks and runtimes.
• pruningPruning is the removal of selected model parameters or computational structures to reduce model size or computation.
• RESTA REST API is a web interface that exposes resources or operations through HTTP requests following REST architectural conventions.
• TensorFlow.jsTensorFlow.js is a JavaScript library for running and developing machine-learning models in browsers and JavaScript environments.
• TensorFlow ServingTensorFlow Serving is a system for serving trained machine-learning models through production prediction APIs.
• TFLiteTensorFlow Lite is a framework for running machine-learning inference efficiently on mobile, embedded, and resource-constrained devices.
• multi-step reasoningReasoning is the process of deriving conclusions or selecting actions from representations, premises, or rules.
• symbolic tensorA symbolic tensor is a tensor-like object that encodes the shape and data type of future data without containing actual data values.
• reuseReuse is the application of an existing module, representation, or operation in multiple contexts instead of reimplementing it.
• GPT modelGPT is a family of autoregressive Transformer language models that generate text by predicting successive tokens from preceding context.
• mixed precisionMixed-precision training is model training that uses numerical values with different floating-point precisions to balance computational efficiency and numerical accuracy.
• hierarchyA JAX tree is a nested structure of named or unnamed lists and array values used as structured function state.
• variableA TensorFlow variable is a mutable tensor-like object that stores modifiable model state and can be updated by assignment.
• concept driftConcept drift is a change over time in the data-generating patterns or relationships that a deployed machine-learning model depends on.
• data annotation infrastructureData annotation infrastructure is the set of processes and software systems for creating, recording, and managing data annotations.
• calibration dataRepresentative data is data whose relevant properties and distribution resemble those of the environment and inputs encountered in production.
• selection biasSampling bias is systematic distortion in measurements caused by a data-collection process that selects examples in relation to the quantity being predicted.
• adaptabilityAdaptability is the ability of a system or agent to adjust its behavior in response to unexpected conditions or new information.
• feature selectionFeature selection is the process of choosing a subset of input features that are useful for a machine-learning task.
• value-centric abstractionValue-centric abstraction is abstraction based on continuous similarity and perceptual or intuitive relationships among representations.
• downsamplingDownsampling is the reduction of a representation's spatial resolution by selecting, aggregating, or skipping input locations.
• splitSplitting is the segmentation of text into a sequence of individual tokens.
• indexingIndexing is the mapping of text tokens to integer positions in a vocabulary.
• text encoderA text encoder is a model that converts textual input into a numerical representation suitable for downstream computation.
• determinismDeterminism is the property that identical inputs and explicit state produce identical computational results.
• global stateGlobal state is mutable information maintained outside a function and implicitly shared across calls.
• jax.jitjax$jit() is a JAX transformation that compiles a stateless function for optimized execution, typically through XLA.
• keyA PRNG key is an explicit data object that identifies the deterministic state used by a pseudo-random number generation operation.
• pseudo-random number generatorA pseudo-random number generation operation deterministically produces values that appear random from an algorithmic seed or key.
• pure functionA pure function produces results solely from its explicit inputs and has no externally observable side effects.
• adamAdam is an adaptive stochastic optimization algorithm that uses estimates of first and second moments of gradients to update parameters.
• transformer decoderA Transformer decoder is an architectural component that generates sequence representations or outputs using attention and feedforward processing.
• transformer encoderA Transformer encoder is an architectural component that transforms input sequence representations using self-attention and feedforward processing.
• stateA neural-network state is an internal representation that carries information from earlier sequence elements to later processing steps.
• activation functionAn activation function transforms a neural-network unit's pre-activation value into its output, often element by element.
• vectorizationData vectorization is the conversion of input data or targets into numerical tensors suitable for neural-network processing.
• hybrid approachA hybrid system is a computational system that combines learned continuous components with discrete algorithmic components.
• underfittingUnderfitting is a modeling condition in which a model is too simple or insufficiently trained to capture patterns needed for good performance.
• interpretabilityInterpretability is the degree to which a machine-learning model's learned representations, predictions, and decisions can be understood by humans.
• batch gradient descentBatch gradient descent updates model parameters using the gradient computed from all available training data.
• masked attentionCausal attention is attention constrained so that each position can use information from past and present positions but not future positions.
• vector programA vector program is a highly nonlinear learned mapping that transforms points within a latent embedding space.
• Word2Vecword2vec is a family of shallow neural models that learn vector representations of words from local context relationships.
• causal language modelA causal language model is a language model that predicts tokens sequentially from preceding context.
• cognitive automationCognitive automation is the encoding and operationalization of human skills and knowledge in computer systems.
• just-in-time compilationJust-in-time compilation is the compilation of program code during execution into an optimized form for subsequent execution.
• confidenceA confidence score is a numerical estimate of how likely a predicted detection is to represent a valid object.
• tf.functiontf.function is a TensorFlow API that traces Python functions into optimized TensorFlow computation graphs and can request compilation.
• visual encoderAn image encoder is a model component that transforms an input image into a lower-dimensional image embedding.
• regular expressionA regular expression is a pattern notation for matching and manipulating sequences of characters.
• stringrstringr is an R package providing tools for consistent string manipulation and pattern matching.
• spatial hierarchyA spatial hierarchy is an organization in which increasingly complex visual patterns are composed from simpler patterns across neural-network layers.
• maskMasking is the exclusion of designated sequence positions from downstream neural-network computations.
• text augmentationText augmentation is the creation of altered or generated textual training examples intended to improve a model's robustness or address data scarcity.
• Modularity-Hierarchy-Reuse formulaThe Modularity-Hierarchy-Reuse formula is an architectural principle of structuring complexity into modules, organizing them hierarchically, and reusing modules across a system.
• image generationImage generation is the computational production of images from learned data distributions or specified inputs.
• vectorA vector is a quantity represented by an ordered collection of components and interpreted geometrically as a directed displacement or point relative to an origin.
• device meshA device mesh is a multidimensional logical arrangement of computational devices organized along named axes.
• LayoutMapLayoutMap is a Keras API class that maps model-variable paths to sharding or replication specifications over a device mesh.
• replicationReplication is the copying of the same variable value to multiple devices along a device-mesh axis.
• tensorflow R packageThe tensorflow R package is an R interface that exposes the TensorFlow Python API and functionality.
• XLAXLA is a compiler for accelerating linear algebra and machine-learning computations across supported hardware.
• ensembleA model ensemble is a collection of predictive models whose outputs are combined to produce a joint prediction.
• stateless function componentA stateless function is a function whose changing computational state is supplied through its arguments and returned explicitly rather than hidden in mutable external state.
• autoencoderAn autoencoder is a neural-network model that learns to encode inputs into representations and decode those representations back into approximations of the inputs.
• channels-first conventionA channels-first image format stores the color-channel axis before the spatial height and width axes.
• callbackA Keras callback is an object invoked at specified points during model training or evaluation to monitor state or perform actions.
• fit methodThe fit method trains a neural network on supplied data by iteratively updating its parameters.
• reticulatereticulate is an R package that provides an interface for calling Python code and managing Python interoperability from R.
• precisionFloating-point precision is the degree of numerical detail representable by a floating-point data format.
• heuristicA heuristic is a method based on manually specified rules rather than rules learned from data or derived from a formal theory.
• self-supervised learningSelf-supervised learning is a machine-learning technique in which training targets are derived from the input data rather than supplied as manual labels.
• AI agentAn intelligent agent is a system capable of adapting its behavior and selecting actions beyond merely executing hardcoded behavioral programs.
• automatonAn automaton is a system that executes predefined instructions or behavioral rules without flexible, general-purpose intelligence.
• samplingSampling is the process of drawing values or instances from a probability distribution or representation space.
• sequence-to-sequence learningSequence-to-sequence learning is the prediction of an output sequence from an input sequence, potentially with different lengths or representations.
• positional embeddingA positional embedding is a representation that encodes an element's location in a sequence and is combined with its content representation.
• datasetA dataset is a collection of data instances organized for analysis or computational learning.
• byte pair encodingByte-pair encoding is an iterative algorithm that builds a vocabulary by repeatedly merging the most frequent adjacent symbol pairs.
• SubWordTokenizerSubWordTokenizer is a book-specific tokenizer that applies learned subword merge rules to character-split text.
• merge ruleA token merge rule specifies an ordered operation for combining adjacent token symbols into a larger symbol.
• shortcut ruleThe shortcut rule is the principle that optimizing a system for a single task metric produces task-specific solutions without necessarily producing general intelligence.
• next-token predictionNext-token prediction is the task of predicting the next token in a sequence given the preceding tokens.
• cross entropy lossCategorical cross-entropy is a multiclass classification loss that measures the negative log-likelihood of the target class under predicted class probabilities and expects categorically encoded labels.
• sigmoidA sigmoid activation function maps arbitrary real-valued inputs to values in the interval from zero to one.
• handwritten digit classificationHandwritten digit classification is the task of assigning an image of a handwritten digit to its corresponding numeric category.
• MNIST datasetMNIST is a dataset of grayscale 28-by-28 pixel images of handwritten digits labeled from 0 through 9.
• image_dataset_from_directoryimage_dataset_from_directory is a Keras utility that constructs an image-input dataset by reading class-organized image files from a directory hierarchy.
• automationAutomation is the execution of tasks by systems according to predefined training, rules, or programs rather than autonomous adaptation.
• carry stateA carry state is an information stream in an LSTM that transports a representation across sequence timesteps for later modification or use.
• SimpleRNN layerA SimpleRNN layer is a Keras recurrent layer that computes sequence representations using basic recurrent state updates.
• input spaceAn input space is the space of possible values that can be supplied as inputs to a model or mathematical function.
• attention scoreAn attention score is a numerical value measuring the relevance of one vector or sequence position to another in an attention calculation.
• reward modelA reward model is a model that assigns a scalar score representing the desirability of a response to an input.
• ranked-preference datasetA ranked-preference dataset is a collection of prompts and candidate responses ordered according to human judgments of response quality.
• reinforcement learning with human feedbackReinforcement learning with human feedback is a training approach that optimizes a model using human preference judgments converted into reward signals.
• cellA grid cell is one rectangular region of a prediction grid to which object-center assignments and predictions are made.
• in-context learningIn-context learning is the adaptation of a model's output behavior from examples or instructions supplied within the current input context without parameter updates.
• temperature forecastingTemperature forecasting is the task of predicting a future temperature value from previous measurements and related observations.
• SAMSegment Anything is a promptable image-segmentation model that generates segmentation masks from images and user-provided prompts.
• Sequential modelA sequential model is a Keras model that represents a simple linear stack of layers with one input and one output.
• classification headA classification layer is a neural-network layer that produces outputs representing predicted class assignments or probabilities.
• rank-4 tensorA rank-4 tensor is a tensor with four dimensions, formed by stacking rank-3 tensors along an additional dimension.
• point promptA point prompt is a spatial point supplied to a segmentation model to indicate a location relevant to the desired segmented region.
• class activation heatmapClass activation map visualization is a technique that produces spatial heatmaps indicating which input locations support a model's predicted class.
• upsamplingUpsampling is the expansion of a representation's spatial resolution by generating or restoring values at additional spatial locations.
• data transformationA data transformation is an operation that changes the form or encoding of data while preserving information relevant to a task.
• matrix-matrix multiplicationmatmul is an operation that computes matrix or batched tensor multiplication.
• low-rank adaptationLow-rank adaptation is a parameter-efficient fine-tuning technique that freezes a model's original weights and trains a low-rank update to selected projections.
• greedy algorithmGreedy search is a sequence-generation strategy that selects the most probable next token at every generation step.
• top-k samplingTop-k sampling selects the next token from only the k highest-scoring candidate tokens and assigns zero probability to the rest.
• torch.nn.Moduletorch.nn.Module is PyTorch's base class for building objects that represent neural-network modules and manage their parameters.
• build()The Keras build method creates a layer's state after receiving the shape of its input.
• combinatorial explosionCombinatorial explosion is the rapid growth of the number of possibilities generated by combining multiple choices or components.
• search programProgram search is the exploration of candidate programs to find one satisfying a specification or objective.
• recurrent dropoutA time-constant dropout mask is a dropout mask whose pattern remains fixed across the timesteps of a sequence.
• dataset_batchdataset_batch is a dataset transformation method that groups individual elements into batches.
• tfdatasetstfdatasets is an R package that provides bindings for constructing and transforming TensorFlow data input pipelines.
• data leakageAn information leak is unintended transfer of information from evaluation data into a model or its development decisions.
• BLEU scoreA BLEU score measures the similarity of machine-translated text to one or more high-quality reference translations using n-gram overlap.
• layer connectivityLayer connectivity is the pattern of directed connections linking neural-network layers and their input or output nodes.
• model subclassingA model subclass is a Keras model class created by extending the Keras Model class and defining its computation in a call method.
• metricA custom metric is a task-specific quantitative measure defined to evaluate success according to particular project goals.
• torch.compile()torch.compile is a PyTorch utility that transforms functions or module instances into compiled versions intended to execute faster.
• bidirectional rnnA bidirectional recurrent neural network is a recurrent architecture that processes a sequence in both chronological and reverse directions and merges the resulting representations.
• dense samplingDense sampling is the collection of observations at sufficiently close locations throughout a relevant space to capture its structure.
• machine learning workflowA machine learning workflow is an organized sequence of activities for defining a problem, preparing data, developing a model, and deploying and maintaining the resulting system.
• locally optimal solutionA local minimum is a point whose function value is no greater than the values at sufficiently nearby points.
• momentumMomentum is an optimization technique that incorporates previous parameter updates into the current update to improve movement through the optimization landscape.
• missing valueA missing value is an absent or unavailable data entry for an expected attribute of an instance.
• model.stateless_call()stateless_call is a Keras model method that computes outputs from explicit inputs and weights while returning updated nontrainable weights.
• optimizer.stateless_apply()stateless_apply is a Keras optimizer method that computes updated parameters and optimizer state from explicit inputs without mutating hidden state.
• denoising modelA denoising model is a model that transforms a noisy representation toward a cleaner representation or generated output.
• vertical partitioningVertical partitioning splits each model layer across all available devices.
• dot-product attentionDot-product attention computes attention relevance scores from the dot products of query-like and source-like vectors and uses them to form a weighted representation.
• SQLA query is the representation whose information need determines which keys receive high attention relevance.
• valueA value is the representation that attention combines after relevance has been computed from queries and keys.
• channels-last conventionThe channels-last convention places an image tensor's color-channel axis after its spatial axes.
• digital imageAn image is visual data represented as a structured arrangement of pixel values.
• rank-5 tensorA rank-5 tensor is a five-axis array commonly used for batches of videos.
• vector descriptionVector data represents each data instance as a one-dimensional vector of numerical attributes.
• videoVideo data represents videos as ordered sequences of image frames.
• deep learning frameworkA deep learning framework is a software system that provides reusable abstractions and computational capabilities for building and training neural networks.
• evaluation protocolAn evaluation protocol is a prescribed procedure for estimating a model's performance on data representative of its intended use.
• compositional generalizationCompositional generalization is the ability to combine learned components to handle novel combinations of functions or situations.
• symbolic AISymbolic AI is an artificial-intelligence approach that represents knowledge explicitly and manipulates it with hand-crafted symbolic rules.
• meta-learning systemA meta-learning system is a machine-learning system that learns strategies or reusable structures for solving multiple learning tasks.
• bag-of-words modelA bag-of-words model represents text as an unordered collection of tokens and assigns learned weights to token presence indicators.
• Inputkeras_input() is a Keras function that creates a symbolic input tensor specifying the shape and data type accepted by a model.
• activationAn activation is the numerical output produced by a neural-network unit or layer for a given input.
• Keras modelThe Keras Model class is an API class representing a trainable neural-network model with methods for training, evaluation, prediction, and export.
• categoryA class is a discrete category assigned to an example in a classification problem.
• language modelingLanguage modeling is the task or technique of learning a probability distribution over sequences of linguistic symbols.
• statistical powerStatistical power in this workflow is a model's ability to achieve performance that exceeds a simple baseline on the task.
• biasBias is a systematic tendency in data, judgments, or model outputs that produces uneven or distorted representations of people or phenomena.
• data availabilityData availability is the extent to which suitable examples and target annotations exist for developing a predictive model.
• RetinaNetRetinaNet is a one-stage object detection architecture designed to detect objects in images using convolutional feature representations and classification and localization heads.
• low-rank approximationA low-rank decomposition represents a matrix transformation as a product of smaller matrices with an intermediate dimension lower than the original matrix dimensions.
• computer visionComputer vision is the field concerned with enabling computers to interpret and analyze visual data such as images and video.
• keras.random.categorical()random_categorical() is a Keras operation that samples category indices from categorical probability distributions.
• compute_losscompute_loss is a Keras model method that computes the configured loss together with applicable regularization losses.
• feature pyramid networkA feature pyramid network is a neural-network architecture that combines semantically rich low-resolution features with spatially detailed high-resolution features at multiple scales.
• downstream taskA downstream task is the target application for which a pretrained model or representation is subsequently adapted or evaluated.
• automatic shape inferenceAutomatic shape inference is the process of determining a layer's input and output shapes from the shapes of tensors supplied to it.
• graph modeGraph mode is a TensorFlow execution mode in which decorated functions are represented and optimized as computation graphs.
• backendA backend engine is a software system that executes the numerical and machine-learning operations required by a higher-level framework.
• problem domainA problem domain is the real-world subject area and context in which a machine learning task is defined and applied.
• generategenerate() is a text-generation method that extends a prompt with tokens sampled according to a configured generation strategy.
• Conv2DTransposeConv2DTranspose is a Keras layer that applies a transposed two-dimensional convolution to increase the spatial resolution of feature representations.
• dropout maskA recurrent dropout mask is a mask that selectively drops values in the recurrent connections or recurrent state of a sequence layer.
• class probability distributionA class probability is a model's estimated probability that an input belongs to a particular class, represented here as a distribution across all classes.
• diversityDiversity is the degree to which models in an ensemble make different errors or rely on different aspects of the data.
• weighted averageA weighted average is an average in which each value contributes according to an associated numerical weight.
• Colossal Clean Crawled CorpusC4 is a large web-derived text dataset used for training and evaluating language models.
• SentencePieceSentencePiece is a library for tokenizing text into subword units without requiring whitespace-delimited preprocessing.
• bfloat16bfloat16 is a 16-bit floating-point data type with an 8-bit exponent and a 7-bit mantissa, providing greater numerical range but lower precision than float16.
• Transformer encoder layerAn encoder block is a composable sequence-processing module that transforms an input sequence while preserving its shape for stacking.
• layer normalizationLayerNormalization is a normalization layer that standardizes activations within each individual sequence or sample independently of other batch elements.
• loss scalingLoss scaling is the technique of multiplying a training loss by a scalar factor to increase the magnitude of gradients during numerical computation.
• adversarial inputAn adversarial example is an input deliberately or naturally modified in a small way that causes a machine-learning model to produce an incorrect output.
• denoisingDenoising is the removal of unwanted or corrupting variation from data to recover a cleaner version.
• box promptA box prompt is an image-segmentation input that approximately bounds the object to be segmented with a rectangular region.
• linearly separableLinearly separable data is data whose classes can be divided by a linear decision boundary.
• squared lossSquared error is a discrepancy measure equal to the square of the difference between a prediction and its target.
• parametric curveA parametric curve is a curve whose points are determined by a finite set of parameters and one or more independent variables.
• rankTensor rank is the number of axes or dimensions of a tensor.
• continuous functionContinuity is the property that sufficiently small changes in a function's input produce sufficiently small changes in its output.
• few-shot learningFew-shot learning is adapting or directing a model to perform a task from only a small number of examples.
• Kullback–Leibler divergenceKullback–Leibler divergence is a measure of how one probability distribution differs from a reference probability distribution.
• sequence generationSequence generation is the production of ordered data elements by a computational model.
• varianceVariance is a statistical measure of the squared variability of values around their mean.
• multilabel classificationMultilabel multiclass classification is a classification task in which each input may belong to multiple categories drawn from a set of more than two classes.
• adaptadapt() is a Keras preprocessing-layer method that learns vocabulary or other data-dependent state from supplied data.
• Abstraction & Reasoning Corpus for Artificial General IntelligenceARC-AGI-1 is the original version of the ARC-AGI benchmark, consisting of abstract reasoning tasks intended to test fluid intelligence.
• mean absolute errorMean absolute error is the average absolute difference between predicted values and their target values.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.