Gradient descent

As explained in Classic Computer Science Problems in Swift: Essential techniques for practicing programmers

Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.

Gradient descent updates neural-network weights to reduce error. Each weight is adjusted using its prior input, the neuron’s delta, and a learning rate; the process repeats as training continues. section 7.2

Updating a network

After calculating deltas, adjust every weight in the hidden and output layers. Train again with another input and expected output, then test the network against known answers. section 7.2

Figure 7.6. The weights of every hidden layer and output layer neuron are updated using the deltas calculated in the previous steps, the prior weights, the prior inputs, and a user-determined learning rate.

See how deltas, prior weights, prior inputs, and the learning rate contribute to updating weights across network layers. section 7.2

How Classic Computer Science Problems in Swift: Essential techniques for practicing programmers covers it

How deep it goes

Describes the weight-update rule and learning-rate role, but stops short of explaining why backpropagation works.

Examples

Uses updates across hidden and output layers, followed by repeated training and testing on known answers.

What you can do after

Apply the update rule to adjust network weights and continue training with new input-output examples.

How it connects

Connects gradient descent with backpropagation, neural networks, error functions, and learning rates.

liveBookmanning.com

Read Classic Computer Science Problems in Swift: Essential techniques for practicing programmers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresGradient descent is an iterative optimization technique that moves a candidate solution in the direction of decreasing objective-function value using gradient information.
• 

  liveBookmanning.com
  Grokking Machine LearningAn iterative optimization technique that changes parameters in the direction of greatest local decrease of an objective function.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningGradient descent is an iterative optimization method that updates parameters in the direction opposite the loss gradient.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionGradient descent is an iterative optimization algorithm that updates parameters in the direction opposite to a function's gradient to reduce a loss or objective.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second Edition
• 

  liveBookmanning.com
  Deep Learning with JAXAn optimization step that updates model parameters using gradients, typically in the opposite direction of the loss gradient.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIGradient descent is an optimization technique that iteratively adjusts parameters in the direction that reduces a differentiable objective function.
• 

  liveBookmanning.com
  Sutskever's ListGradient descent is an iterative optimization method that changes parameters in the direction opposite the gradient of a loss function.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA parameter update step computed from gradients to reduce model error.
• 

  liveBookmanning.com
  Data Without LabelsGradient descent is an iterative optimization technique that updates parameters in the direction of decreasing objective-function value.
• 

  liveBookmanning.com
  Math and Architectures of Deep Learning
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionGradient descent is an optimization technique that iteratively changes parameters in the direction that reduces an objective function.
• 

  liveBookmanning.com
  How Large Language Models WorkGradient descent is an iterative optimization algorithm that adjusts parameters in the direction that reduces a differentiable objective function.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA gradient update, or gradient step, changes model parameters using the gradient of an optimization objective.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionGradient descent is an iterative optimization technique that updates parameters in the direction opposite the objective gradient.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsAn iterative optimization algorithm that changes variables in the direction of the negative gradient to reduce a differentiable objective function.

Compare how 17 books explain gradient descent →

Related concepts

• gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
• stochastic gradient descentincludes · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• neural networkused by · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.
• regularizationuses · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
• newton-raphson methodcontrasts with · The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.

This explainer was generated by Manning's knowledge graph from the text of Classic Computer Science Problems in Swift: Essential techniques for practicing programmers. The section links point to the parts of the book it draws on.