How deeply does Math and Architectures of Deep Learning cover partial derivative?

Defines coordinate-wise change, gives a quadratic example, and connects partial derivatives to total change and gradient direction.

Partial derivative

As explained in Math and Architectures of Deep Learning

The derivative of a multivariable function with respect to one variable while holding the others constant, here squared and accumulated by AdaGrad.

A partial derivative measures how a multivariable function changes when exactly one variable changes while all others remain constant. For the loss function L(w0, w1) = 2w0² + 3w1², the partial derivatives isolate each parameter’s contribution to loss change. Combining these contributions estimates the total change when several parameters move together. section 3.3.1

A quadratic loss surface

At P = (w0, w1) = (3, 4), the loss is 66 and the gradient is [12, 24]ᵀ. Moving in the opposite direction, [-12, -24]ᵀ, produces the greatest local reduction in loss. This illustrates how individual partial derivatives combine into a direction for changing parameters. section 3.3.1

Figure 3.4 δL in terms of δw in one dimension, illustrated with two example curves: a straight line and a parabola. In general, δL = (dL/dw) δw. To decrease loss, δw must have the opposite sign of the derivative dL/dw. In (a), this implies we always have to move left (decrease w) to decrease L. In (b), if we are in the left half (e.g., point Q), the derivative is negative, and we have to move to the right to decrease L. But if we are in the right half, the derivative is positive, and we have to move to the left to decrease L. Geometrically, this is equivalent to following the tangent “downward.”

(a) Line: L(w) = 2w + 1, dL/dw = m

(b) Parabola: L(w) = w², dL/dw = 2w

Use the line-and-parabola comparison to connect one-variable derivative direction with the multivariable idea of changing parameters to decrease loss. section 3.3.1

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Geometry for ProgrammersA partial derivative is the derivative of a multivariable function with respect to one variable while the other variables are held constant.
liveBook manning.com

Advanced Algorithms and Data StructuresA partial derivative is the derivative of a multivariable function with respect to one coordinate while the other coordinates are held constant.

Compare how 3 books explain partial derivative →

Related concepts

gradient descentused by · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
momentumequivalent to · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
gradientequivalent to · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
tangent vectorproduces · A tangent vector is a vector of first-order derivative values representing sensitivity or instantaneous direction under an input perturbation.
derivativekind of · A derivative is a mathematical quantity that measures the instantaneous rate of change of a function with respect to an input variable.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.