Derivative

As explained in Deep Learning with PyTorch, Second Edition

A quantity describing how a function changes with respect to an input or parameter.

A derivative measures how a quantity changes when its input or a model parameter changes. For a loss function, it gives the direction and rate of change at an infinitesimally small neighborhood, avoiding the roughness of probing with a fixed step. section 5.4.2

In learning, calculate the derivative of the loss with respect to each parameter. Collecting those individual derivatives produces the gradient, which describes how the loss changes across the model’s parameters. section 5.4.2 section 5.1

For a linear model and mean squared loss, the calculation follows the chain rule:

• Differentiate the loss with respect to each prediction. For mean squared error, this produces 2 × (prediction − target) divided by the number of data points. section 5.4.2
• Differentiate the model with respect to its parameters. For model = weight × input + bias, the derivatives are the input for the weight and 1 for the bias. section 5.4.2
• Multiply corresponding derivatives and sum across data points to obtain the loss derivatives for the weight and bias. section 5.4.2

From derivatives to a gradient

With model(t_u, w, b) = w × t_u + b and a mean squared loss, compute the loss derivative for every prediction, multiply it by t_u for the weight derivative and by 1 for the bias derivative, then stack the two sums into a gradient. section 5.4.2

Figure 5.6 Comparing numerical vs. analytical gradient calculation. The numerical approach (arrow through points) estimates the gradient using finite differences between discrete points, while the analytical approach (arrow along the curves) calculates the exact gradient at a specific point.

Compare finite differences between discrete points with the analytical derivative evaluated at a specific point. section 5.4.2

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Derives loss and linear-model derivatives with the chain rule, then assembles them into a two-parameter gradient.

Examples

Demonstrates derivatives with a linear model, mean squared loss, weight and bias parameters, and PyTorch-style gradient functions.

What it assumes

Builds from a simple linear model and squared loss before applying the mechanics to more complicated models.

What you can do after

Apply it to calculate a model’s gradient analytically instead of estimating change through repeated finite-difference evaluations.

How it connects

Connects derivatives to gradients, chain rule, loss functions, model parameters, finite differences, and PyTorch.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Geometry for ProgrammersA derivative measures the rate of change of a function with respect to an input variable.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA derivative is a mathematical quantity that measures the instantaneous rate of change of a function with respect to its input.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA derivative is a measure of how a function's output changes in response to an infinitesimal change in an input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe mathematical process of computing derivatives, used here to support gradient-based learning.
• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresA derivative measures the instantaneous rate at which a function changes with respect to its input.

Compare how 6 books explain derivative →

Related concepts

• differentiationproduced by · Differentiation is the mathematical operation of determining a function’s rate of change with respect to its inputs.
• gradient descentused by · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• slopemeasures · Slope is the rate at which a dependent quantity changes relative to an independent quantity, often represented by a line’s gradient or a coefficient in a linear relationship.
• backpropagationused by · Backpropagation is a procedure for computing derivatives of an output or loss with respect to inputs and parameters by applying the chain rule backward through a computational graph.
• gradientincludes · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
• second derivativeincludes · A second derivative is the derivative of a function’s first derivative with respect to the same variable, describing how its rate of change varies.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.