Gradient descent

As explained in Grokking AI Algorithms, Second Edition

Gradient descent is an optimization technique that iteratively changes parameters in the direction that reduces an objective function.

Gradient descent is an optimization technique that repeatedly changes model weights to reduce a cost function. It uses the slope at the current point to choose a direction and step size: move opposite the gradient, with the learning rate controlling the step. section 9.8 section 9.26

A neural network training cycle uses gradient descent like this:

• Evaluate the network and calculate its error or cost. section 9.8
• Use backpropagation and the Chain Rule to calculate each weight’s gradient. section 9.26
• Update every weight in the direction that lowers cost, scaling the change by the learning rate. section 9.26
• Repeat the process across the training data, while watching for overfitting from training too long. section 9.8

Finding the valley

Imagine standing on a dark, hilly cost landscape. A positive slope means moving forward increases error, so move backward; a negative slope means moving forward decreases error, so move forward. A flat slope may indicate that the minimum is near. In a diffusion model, the same pattern updates weights so predicted noise becomes closer to actual noise over many training steps. section 9.8 section 12.23

Figure 9.25 Example of adjusting a weight by using gradient descent

Visualizes an individual weight adjustment using gradient descent. section 9.8

How Grokking AI Algorithms, Second Edition covers it

How deep it goes

Explains slope intuition, learning-rate updates, backpropagation, and Chain Rule calculations without pursuing deeper optimization theory.

Examples

Uses a hilly-landscape analogy, weight-versus-cost figures, ANN updates, and diffusion-model noise prediction.

What it assumes

Builds derivative intuition from basic examples while assuming familiarity with neural-network weights and training.

What you can do after

Apply it to update neural-network weights and tune diffusion models toward more accurate predictions.

How it connects

Connects derivatives, cost functions, learning rates, backpropagation, Chain Rule, overfitting, and diffusion models.

liveBookmanning.com

Read Grokking AI Algorithms, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresGradient descent is an iterative optimization technique that moves a candidate solution in the direction of decreasing objective-function value using gradient information.
• 

  liveBookmanning.com
  Grokking Machine LearningAn iterative optimization technique that changes parameters in the direction of greatest local decrease of an objective function.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningGradient descent is an iterative optimization method that updates parameters in the direction opposite the loss gradient.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionGradient descent is an iterative optimization algorithm that updates parameters in the direction opposite to a function's gradient to reduce a loss or objective.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second Edition
• 

  liveBookmanning.com
  Deep Learning with JAXAn optimization step that updates model parameters using gradients, typically in the opposite direction of the loss gradient.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIGradient descent is an optimization technique that iteratively adjusts parameters in the direction that reduces a differentiable objective function.
• 

  liveBookmanning.com
  Sutskever's ListGradient descent is an iterative optimization method that changes parameters in the direction opposite the gradient of a loss function.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA parameter update step computed from gradients to reduce model error.
• 

  liveBookmanning.com
  Data Without LabelsGradient descent is an iterative optimization technique that updates parameters in the direction of decreasing objective-function value.
• 

  liveBookmanning.com
  Math and Architectures of Deep Learning
• 

  liveBookmanning.com
  How Large Language Models WorkGradient descent is an iterative optimization algorithm that adjusts parameters in the direction that reduces a differentiable objective function.
• 

  liveBookmanning.com
  Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA gradient update, or gradient step, changes model parameters using the gradient of an optimization objective.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionGradient descent is an iterative optimization technique that updates parameters in the direction opposite the objective gradient.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsAn iterative optimization algorithm that changes variables in the direction of the negative gradient to reduce a differentiable objective function.

Compare how 17 books explain gradient descent →

Related concepts

• gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
• stochastic gradient descentincludes · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• neural networkused by · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.
• regularizationuses · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
• newton-raphson methodcontrasts with · The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.