liveBook · Manning

Gradient

As explained in Timeless Algorithms, The Foundational Ideas

A gradient is a vector of partial derivatives that describes the direction and rate of change of a function with respect to its inputs.

A gradient records how the loss changes as a parameter changes. For a parameter θ, ∂L/∂θ gives the direction and rate of that change. Together, these derivatives show how sensitive the loss is to the network’s weights and biases. section 11.2.4

In neural-network training, gradients serve two distinct roles:

  • Backpropagation uses the chain rule to calculate gradients by tracing the loss backward through the network. It computes sensitivities but does not change parameters. section 11.2.4
  • Gradient descent uses those gradients to update weights and biases. The learning rate controls the size of each adjustment. section 11.2.5

A backward-pass calculation

For a predicted probability of 0.68 and an actual outcome of 1, the output error signal is −0.32. Multiplying it by a hidden-node output of 0.75 gives an output-weight gradient of −0.24. These values describe loss sensitivity; they are not parameter updates. section 11.3.4

θnew=θold-η∂L∂θ

The update rule subtracts the learning-rate-scaled gradient from the current parameter; the gradient informs the direction, while the learning rate sets the adjustment size. section 11.2.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines gradients as loss sensitivities and works through chain-rule calculations for output- and hidden-layer parameters, distinguishing calculation from updates.
Examples
Calculates output- and hidden-layer gradients for a network with a predicted probability of 0.68 and an actual outcome of 1.
What it assumes
Uses the chain rule from calculus; no broader prerequisites or from-zero treatment are specified.
What you can do after
Evaluate how weights and biases affect loss by tracing derivatives backward, then use those sensitivities to guide gradient-descent updates.
How it connects
Connects loss functions and the chain rule to backpropagation and gradient descent in neural-network training.

Also explained in

Compare how 27 books explain gradient →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.