An iterative optimization algorithm that changes variables in the direction of the negative gradient to reduce a differentiable objective function.
Gradient descent is an iterative optimization algorithm that minimizes a continuous differentiable function by moving in the direction of its negative gradient—the steepest local downward slope. It is widely used to train machine-learning models and make predictions. section 4.3.1
Key properties distinguish gradient descent from hill climbing:
Minimizes an objective by hiking down toward the bottom of a valley, whereas gradient ascent maximizes an objective by following the positive gradient. section 4.3.1
Usually converges to a global minimum for a convex function when the learning rate is properly chosen. section 4.3.1
Requires differentiability, unlike hill climbing, which suits discrete optimization problems and can become trapped in local optima. section 4.3.1
How Optimization Algorithms: AI techniques for design, planning, and control problems covers it
How deep it goes
Defines iterative negative-gradient movement and notes convexity and learning-rate conditions, but does not provide implementation steps.
Examples
Uses hiking down to the bottom of a valley as an intuition for following the steepest negative slope.
What it assumes
Assumes familiarity with continuous differentiable functions, local slope, convexity, and learning rate.
What you can do after
Evaluate optimization methods for differentiable objectives and distinguish them from hill climbing.
How it connects
Contrasts gradient descent with hill climbing and gradient ascent, and connects convergence to convex functions and learning rate.
Advanced Algorithms and Data StructuresGradient descent is an iterative optimization technique that moves a candidate solution in the direction of decreasing objective-function value using gradient information.
Grokking Machine LearningAn iterative optimization technique that changes parameters in the direction of greatest local decrease of an objective function.
Ensemble Methods for Machine LearningGradient descent is an iterative optimization method that updates parameters in the direction opposite the loss gradient.
Deep Learning with R, Third EditionGradient descent is an iterative optimization algorithm that updates parameters in the direction opposite to a function's gradient to reduce a loss or objective.
Sutskever's ListGradient descent is an iterative optimization method that changes parameters in the direction opposite the gradient of a loss function.
Data Without LabelsGradient descent is an iterative optimization technique that updates parameters in the direction of decreasing objective-function value.
Grokking AI Algorithms, Second EditionGradient descent is an optimization technique that iteratively changes parameters in the direction that reduces an objective function.
How Large Language Models WorkGradient descent is an iterative optimization algorithm that adjusts parameters in the direction that reduces a differentiable objective function.
Bayesian Optimization in ActionGradient descent is an iterative optimization technique that updates parameters in the direction opposite the objective gradient.
gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
stochastic gradient descentincludes · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
neural networkused by · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.
regularizationuses · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
newton-raphson methodcontrasts with · The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.
This explainer was generated by Manning's knowledge graph from the text of Optimization Algorithms: AI techniques for design, planning, and control problems. The section links point to the parts of the book it draws on.
