Taylor series

As explained in Math and Architectures of Deep Learning

A multivariable Taylor expansion that approximates a function near a point and yields quadratic terms used in optimization arguments.

A Taylor series approximates a smooth function near a chosen point using the function’s value and derivatives there. For a loss function, it describes how a parameter displacement changes the loss. The infinite expansion is exact only when all terms are included, but higher-order terms usually become negligible for small displacements. section 3.4 section 3.4.1

The approximation becomes progressively richer as terms are added:

• The first-order approximation keeps the gradient term and behaves like the tangent line; for very small displacements, it captures the main change. section 3.4.1 section 3.4.2
• The second-order approximation adds a quadratic displacement term governed by the Hessian matrix, improving accuracy for somewhat larger displacements. section 3.4 section 3.4.2
• In convex functions, a positive second derivative makes the quadratic approximation lie above or equal to the linear approximation locally. section 3.7.3

Exponential near zero

For the exponential function near x = 0, every derivative has value 1 at the expansion point. Its Taylor series therefore consists of terms whose successive powers of x become smaller near zero, making a short truncation useful. section 3.4.1

Equation 3.8

Use the multidimensional expansion figure to connect a parameter displacement with the gradient term and Hessian-based quadratic term. section 3.4.2

The expansion explains gradient descent: after dropping negligible quadratic and higher-order terms, the largest local decrease occurs along the negative gradient. The learning rate controls the step size; larger values move faster but may overshoot the minimum. section 8.4.3

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines one- and multidimensional expansions, explains gradient and Hessian terms, and stops short of routinely computing higher-order terms.

Examples

Uses an exponential expansion at zero, tangent and quadratic curve interpretations, and parameter displacements in loss landscapes.

What it assumes

Builds from one-variable derivatives before extending the idea to gradients and Hessian matrices.

What you can do after

Evaluate local loss changes and understand why negative-gradient steps provide the fastest infinitesimal descent direction.

How it connects

Connects derivatives, gradients, Hessian matrices, convexity, gradient descent, and learning rates.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Geometry for ProgrammersA Taylor series is a power series whose coefficients are determined by the derivatives of a function at a specified center.
• 

  liveBookmanning.com
  Julia as a Second LanguageA Taylor series is a power series representing a function as an expansion around a specified point using its derivatives.

Compare how 3 books explain Taylor series →

Related concepts

• Maclaurin seriesincludes · A Maclaurin series is a power series representing a function using its derivatives evaluated at zero.
• sine functionuses · The sine function is a periodic trigonometric function that maps a real number or angle to a value between −1 and 1.
• derivativeuses · A derivative is a mathematical quantity that measures the instantaneous rate of change of a function with respect to an input variable.
• trigonometryimplements · Trigonometry is the branch of mathematics studying relationships among angles, lengths, geometric ratios, coordinates, and periodic functions.
• exponentialused by · An exponential function is a function in which a fixed positive base is raised to a variable exponent.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.