Gradient
A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
Deep Learning with R, Third Edition
A gradient is the tensor-valued derivative of a function whose inputs are tensors, with each coefficient indicating how the output changes with respect to one input coefficient.
- How it is used
- Uses gradients to guide neural network weight updates and to investigate model behavior through input images.
- How deep it goes
- Explains gradient shape and local meaning, then implements automatic differentiation and weight updates rather than backpropagation from scratch.
- Examples
- Uses weight perturbations, a handwritten-digit training step, and a PyTorch linear classifier to illustrate gradients.
- What you can do after
- Apply gradients using TensorFlow GradientTape and Keras SGD to update model weights.
Read how Deep Learning with R, Third Edition explains gradient →
Math and Architectures of Deep Learning
The vector of partial derivatives indicating the local direction of steepest increase of a function, used here to choose the steepest ascent or descent direction.
- How it is used
- Guides iterative loss minimization by selecting parameter-update directions for classifiers and neural networks.
- How deep it goes
- Defines gradients from partial derivatives, derives local change and geometry, and connects first-order Taylor approximation to the Hessian.
- Examples
- Uses one-dimensional lines and parabolas, quadratic loss surfaces, level contours, spheres, cylinders, and noisy minibatch gradients.
- What you can do after
- Evaluate/understand why negative loss gradients define local descent directions and how training updates weights and biases.
Read how Math and Architectures of Deep Learning explains gradient →
Build AI Drug Discovery Pipelines
A gradient is a vector containing the partial derivatives of a scalar function with respect to its variables and pointing toward steepest increase.
- How it is used
- Uses gradients to direct parameter updates that reduce prediction loss in solubility and neural-network models.
- How deep it goes
- Explains gradient descent and backpropagation through computational graphs, then details gradient calculation and updates in a training loop.
- Examples
- Illustrates gradients with EGFR activity prediction and a logistic regression calculation with trainable weights and bias.
- What you can do after
- Apply it via PyTorch to compute parameter gradients and update a model during training.
Read how Build AI Drug Discovery Pipelines explains gradient →
Natural Language Processing in Action, Second Edition
A gradient is the vector of partial derivatives that indicates how a function changes with respect to its parameters or inputs.
- How it is used
- Guides neural networks in assigning weight updates that reduce prediction loss during training.
- How deep it goes
- Descends from weight-specific slopes to learning-rate selection and serial backpropagation through recurrent token sequences.
- Examples
- Uses CNN parameter updates and RNN token-by-token error propagation to show how gradients behave in different architectures.
- What you can do after
- Apply gradients through backpropagation and optimizers to understand or adjust neural-network training.
Read how Natural Language Processing in Action, Second Edition explains gradient →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
A gradient is a quantity indicating the direction and rate of greatest local increase of an objective with respect to parameters.
- How it is used
- Guides iterative weight and prediction updates for SGD, regularized regression, and gradient boosting.
- How deep it goes
- Defines gradients through partial derivatives and steepest descent, then applies them to batches, regularization, and logit-based boosting without a full derivation.
- Examples
- Uses an error-landscape descent visualization, out-of-core logistic regression, and probability histograms for gradient boosting classification.
- What you can do after
- Apply it via SGDClassifier and gradient-boosting implementations to update models from loss-driven corrections.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains gradient →
Geometry for Programmers
A gradient is a vector of partial derivatives describing the direction and rate of greatest increase of a multivariable function.
- How it is used
- Guides contour vertices toward target isolines and supplies derivative constraints for smoothing segments.
- How deep it goes
- Defines gradients through partial derivatives, then applies their direction and magnitude to fitting, sharp features, and cubic tangents.
- Examples
- Uses a two-variable polynomial, SDF contour shifts, interpolated threshold images, and normalized cubic tangent vectors.
- What you can do after
- Apply it to move contour vertices, derive cubic endpoint constraints, and control tangent-vector scale.
Grokking Machine Learning
A vector of partial derivatives that indicates how a multivariable function changes with respect to its parameters.
- How it is used
- Guides neural-network training by supplying derivatives used to reduce loss.
- How deep it goes
- Defines derivatives for each weight and bias and gives the update procedure; detailed backpropagation mathematics is deferred.
- Examples
- Uses neural-network loss derivatives for weights and biases, followed by small opposite-gradient updates.
- What you can do after
- Apply gradient descent by calculating the loss and its gradient, then stepping a small amount in the opposite direction.
Deep Learning with JAX
The derivative of a function with respect to its inputs, used here in the optimization sense for training neural networks.
- How it is used
- Uses gradients to train neural networks by minimizing a loss through repeated parameter updates.
- How deep it goes
- Explains autodiff, updates, options, derivative modes, Jacobians, and Hessians, while deferring detailed framework comparisons elsewhere.
- Examples
- Demonstrates gradients with neural-network loss updates, a scalar function, a loss landscape, and TensorFlow gradient tapes.
- What you can do after
- Apply gradients via JAX grad and value_and_grad to implement parameter updates and monitor training loss.
Ensemble Methods for Machine Learning
A gradient is the vector of partial derivatives of a scalar function with respect to its input variables.
- How it is used
- Guides optimization by supplying a local direction for reducing a loss function.
- How deep it goes
- Explains first-order slope, descent direction, step selection, convergence, and a working implementation, while deferring second-order curvature.
- Examples
- Uses the Branin function, surface and contour plots, descent paths, and a NumPy/SciPy implementation.
- What you can do after
- Builds a simple gradient-descent optimizer and interprets its path toward local minima.
Read how Ensemble Methods for Machine Learning explains gradient →
Inside Deep Learning: Math, Algorithms, Models
The vector of partial derivatives of a function with respect to its variables or model parameters.
- How it is used
- Uses gradients to minimize loss functions and update neural-network parameters during training.
- How deep it goes
- Explains derivatives, parameter-wise notation, PyTorch computation, gradient descent, and vanishing gradients without deriving automatic differentiation internals.
- Examples
- Demonstrates f(x) = (x − 2)², PyTorch backward(), neural-network training loops, and tanh or sigmoid derivative plots.
- What you can do after
- Apply it via PyTorch automatic differentiation to compute derivatives and understand parameter updates during model training.
Read how Inside Deep Learning: Math, Algorithms, Models explains gradient →
Grokking AI Algorithms, Second Edition
A gradient is a vector or matrix of partial derivatives that describes how a function changes with respect to multiple parameters.
- How it is used
- Guides weight updates that reduce neural-network training loss.
- How deep it goes
- Explains derivative signs, the chain rule, a layer’s gradient matrix, and a numerical update; leaves full gradient-update code as a placeholder.
- Examples
- Uses a weight update from 0.5 to 0.505 and a diffusion-model loss comparing predicted and actual noise.
- What you can do after
- Evaluate how a gradient’s sign and learning rate determine a weight update.
Read how Grokking AI Algorithms, Second Edition explains gradient →
Deep Learning with Python, Third Edition
The generalization of a derivative to a tensor-valued function, representing the local rate and direction of change with respect to each input component.
- How it is used
- Uses gradients to guide weight updates and to modify inputs during ConvNet filter visualization.
- How deep it goes
- Explains tensor gradients component by component and relates them to parameter updates and backpropagation.
- Examples
- Illustrates a weight-matrix loss gradient and gradient ascent on an image for ConvNet filter visualization.
- What you can do after
- Evaluate/understand how gradient direction and magnitude guide weight updates and input changes.
Read how Deep Learning with Python, Third Edition explains gradient →
Sutskever's List
A gradient is the vector of partial derivatives describing how a function changes with respect to its parameters.
- How it is used
- Guides neural-network weight updates by supplying a local direction for reducing loss.
- How deep it goes
- Explains partial derivatives, backpropagation products, optimization geometry, and update settings without deriving curvature methods.
- Examples
- Uses a foggy-mountain hiker, mini-batch updates, AlexNet settings, and narrow optimization valleys.
- What you can do after
- Apply gradient-based training through mini-batch SGD, momentum, learning-rate schedules, and initialization choices.
Deep Learning with PyTorch, Second Edition
A vector of derivatives describing a function’s change with respect to its inputs or parameters.
- How it is used
- Uses gradients to guide parameter updates that reduce prediction error.
- How deep it goes
- Derives a linear model’s loss gradient with the chain rule, then implements automatic differentiation and training updates.
- Examples
- Works through thermometer calibration with a linear model and inspects gradients in a sequential neural network.
- What you can do after
- Apply gradients via PyTorch autograd and an optimizer to train differentiable models.
Read how Deep Learning with PyTorch, Second Edition explains gradient →
Advanced Algorithms and Data Structures
A gradient is the vector of partial derivatives of a multivariable scalar function, indicating the direction of greatest local increase.
- How it is used
- Uses local slope information to reduce cost functions during iterative optimization.
- How deep it goes
- Defines partial derivatives, geometric direction, coordinate updates, stopping conditions, and differentiability requirements without formal proofs.
- Examples
- Illustrates gradients with marble movement, one-variable curves, two-variable surfaces, and graph-drawing optimization.
- What you can do after
- Apply it via gradient descent to iteratively seek low-cost solutions in differentiable optimization problems.
Read how Advanced Algorithms and Data Structures explains gradient →
Data Without Labels
A gradient is a vector of partial derivatives that indicates how a scalar function changes with respect to multiple variables.
- How it is used
- Guides neural-network training by indicating parameter updates that reduce prediction loss.
- How deep it goes
- Explains steepest descent, batch variants, backward derivatives, and chain-rule calculation, but stops short of implementation code.
- Examples
- Uses batch, stochastic, and mini-batch updates, neural-network backpropagation, and Boltzmann energy minimization as examples.
- What you can do after
- Apply it via gradient-descent variants and backpropagation to understand iterative loss-reducing weight updates.
How Large Language Models Work
A gradient is a vector of derivatives indicating how an objective function changes as its parameters change.
- How it is used
- Explains how training algorithms use error measurements to improve neural-network parameters.
- How deep it goes
- Defines direction and amount of parameter change, then traces repeated updates while stopping short of mathematical derivation.
- Examples
- Uses a ball rolling downhill, loss landscapes, training inputs, expected outputs, and parameter-update diagrams.
- What you can do after
- Evaluate how gradients guide neural-network training and why repeated updates may still miss the best solution.
Building Specialized AI Systems
A gradient is a vector of partial derivatives indicating how a model objective changes with respect to model parameters.
- How it is used
- Guides parameter updates during model training and task-specific fine-tuning by turning loss measurements into adjustment directions.
- How deep it goes
- Explains parameters, loss, descent intuition, and the training-loop backward step, but stops short of derivative calculations.
- Examples
- Uses foggy-hill navigation, sentiment probabilities, a single-weight figure, and a basic training-loop listing.
- What you can do after
- Understand how loss, backpropagation, and optimizer steps adjust model parameters during training.
Read how Building Specialized AI Systems explains gradient →
Timeless Algorithms, The Foundational Ideas
A gradient is a vector of partial derivatives that describes the direction and rate of change of a function with respect to its inputs.
- How it is used
- Supplies parameter-sensitivity signals that an optimization method uses to adjust neural-network weights and biases.
- How deep it goes
- Defines gradients as loss sensitivities and works through chain-rule calculations for output- and hidden-layer parameters, distinguishing calculation from updates.
- Examples
- Calculates output- and hidden-layer gradients for a network with a predicted probability of 0.68 and an actual outcome of 1.
- What you can do after
- Evaluate how weights and biases affect loss by tracing derivatives backward, then use those sensitivities to guide gradient-descent updates.
Read how Timeless Algorithms, The Foundational Ideas explains gradient →
CUDA for LLMs
A gradient is a collection of derivatives describing how a function changes with respect to its parameters.
- How it is used
- Tracks how network parameters and intermediate values affect loss during backpropagation.
- How deep it goes
- Works from scalar chain-rule propagation through matrix gradients and layer-specific formulas for linear, ReLU, and softmax-cross-entropy operations.
- Examples
- Uses a scalar multiplication-addition-square chain and formulas for linear layers, ReLU, and softmax-cross-entropy.
- What you can do after
- Understand how to calculate weight, bias, input, ReLU, and combined softmax-cross-entropy gradients.
Quantization and Fast Inference
A gradient is a vector of derivatives describing how a function changes with respect to its inputs or parameters.
- How it is used
- Preserves learning signals during quantization-aware training when rounding would otherwise block weight updates.
- How deep it goes
- Explains rounding derivatives, STE and clipping, then compares gradient survival under per-tensor and per-channel quantization.
- Examples
- Uses a four-value rounding example and an INT4 gradient-survival comparison on ResNet-18 conv1.
- What you can do after
- Diagnose why rounding blocks QAT gradients and compare quantization granularity by its effect on gradient survival.
Read how Quantization and Fast Inference explains gradient →
Applied Reinforcement Learning
A gradient is a vector of partial derivatives that indicates how a scalar function changes with respect to multiple variables.
- How it is used
- Supports stable learning in deep reinforcement-learning networks.
- How deep it goes
- Connects gradients to stable learning and displays a chain-rule loss-derivative expression across network layers.
Grokking Bayes
A gradient is the vector of partial derivatives that indicates the direction of greatest local increase of a differentiable function.
- How it is used
- Optimizes variational approximations by updating distribution parameters toward a better ELBO.
- How deep it goes
- Defines the direction of steepest decrease and gives an iterative update-and-convergence procedure, without developing technical details.
- Examples
- Uses μ and σ of a Normal approximating distribution and the ELBO as the optimization objective.
- What you can do after
- Apply gradient descent or ascent to iteratively optimize parameters using objective-function gradients.
Reinforcement Learning from Human Feedback
A gradient is a vector of derivatives indicating how a scalar function changes with respect to its parameters.
- How it is used
- Translates reward or preference signals into parameter updates for language-model policies.
- How deep it goes
- Derives the DPO gradient through sigmoid and logarithm identities, then compares policy-gradient estimators and stability choices.
- Examples
- Uses generated completions, reward scores, chosen-versus-rejected pairs, token-level margins, and clipped policy objectives.
- What you can do after
- Apply it via policy-gradient and DPO objectives to update models toward higher reward or stronger preference margins.
Read how Reinforcement Learning from Human Feedback explains gradient →
Machine Learning Algorithms in Depth
A gradient is a vector of partial derivatives that indicates how a scalar function changes with respect to its variables.
- How it is used
- Guides parameter updates while training neural networks to minimize loss.
- How deep it goes
- Explains gradient-based update directions, learning rates, momentum, Nesterov momentum, and adaptive optimization methods.
- Examples
- Uses a CIFAR-100 convolutional neural network and compares cross-entropy loss and validation accuracy across optimizers.
- What you can do after
- Apply it via stochastic gradient descent or related optimizers to train convolutional neural networks.
Read how Machine Learning Algorithms in Depth explains gradient →
Distributed Machine Learning Patterns
A gradient is a vector of partial derivatives that indicates how a function changes with respect to its parameters.
- How it is used
- Uses calculated gradients to update model parameters across parallel workers during distributed model training.
- How deep it goes
- Explains gradients through worker computation and communication, but stops short of mathematical derivation or optimization internals.
- Examples
- Uses parameter-server architecture, worker-only replication, and Fashion-MNIST training to illustrate gradient calculation and updating.
- What you can do after
- Apply it via distributed training by assigning data partitions to workers and coordinating their gradient-based model updates.
Read how Distributed Machine Learning Patterns explains gradient →
Transformers in Action
A gradient is a vector of partial derivatives indicating how a function changes with respect to its parameters.
- How it is used
- Guides gradient descent updates during model training by indicating how parameters should change to reduce loss.
- How deep it goes
- Explains loss-relative parameter updates and training controls, without deriving partial derivatives or detailing their implementation.
- Examples
- Contrasts gradient-descent convergence with learning rates of 0.1 and 0.01.
- What you can do after
- Apply it via gradient descent to understand parameter updates and choose learning-rate, batch-size, and epoch settings.
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- Machines that ThinkIn early access (MEAP)
- Designing Deep Learning Systems: A guide for software engineers
- Crack Any Codebase with AIIn early access (MEAP)
- Grokking Machine Learning, Second EditionIn early access (MEAP)
What to know first
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
Related concepts
- backpropagationproduced by · Backpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- gradient descentused by · Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
- gradient normmeasured by · A gradient norm is a scalar measure of the magnitude of a function's gradient vector.
- maximummeasured by · A maximum is the greatest element in a collection or range under a specified ordering.
Questions about gradient
What is gradient?
A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
Which Manning books explain gradient?
Deep Learning with R, Third Edition, Math and Architectures of Deep Learning, Build AI Drug Discovery Pipelines, Natural Language Processing in Action, Second Edition, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Geometry for Programmers, Grokking Machine Learning, Deep Learning with JAX and 19 more.
What should I know before learning gradient?
Manning books that teach gradient usually introduce objective function, language model, parameter and backpropagation first.
What is gradient also called?
Gradient is also called grad.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


























