Loss surface

As explained in Sutskever's List

A loss landscape is the function relating a model's parameter values to its loss across parameter space.

A loss surface maps model parameter values to their loss. Before AlexNet, researchers often pictured it as terrain riddled with poor local minima and saddle points, making deep-network training seem unstable. That picture overlooked how loss surfaces behave in high-dimensional spaces. section 2.4 section 2.5.4

Figure 2.2 The challenge of optimizing functions shaped like long, narrow valleys. The contour lines represent a valley, with the optimal direction of progress indicated by an arrow running along the valley’s base. Smaller arrows represent steps taken by gradient descent: the left diagram shows large steps oscillating inefficiently across the valley, while the right diagram shows small steps making slow progress along the base. The “pathological” nature does not arise from high or low curvature, but from the combination of high curvature across the valley and low curvature along it. Used with permission granted by the author (James Martens).

A long, narrow valley shows why gradient descent can make inefficient progress: large steps oscillate across the valley, while smaller steps move slowly along its base. section 2.4

AlexNet’s mini-batch gradient averaging further smoothed irregularities in the loss landscape, supporting more stable optimization. The landscape metaphor helps explain why training behavior depends not only on local minima but also on dimensionality and the shape of the terrain. section 2.5.4 section 2.4

How Sutskever's List covers it

How deep it goes

Explains terrain metaphors, valley curvature, high-dimensional behavior, and mini-batch smoothing without formal analysis.

Examples

Uses a long, narrow valley to contrast oscillating large steps with slow progress along the valley’s base.

What you can do after

Understand why high-dimensional loss surfaces and mini-batch gradient averaging complicate the expectation that optimization gets trapped in poor minima.

How it connects

Connects loss surfaces to gradient descent, local minima, saddle points, and mini-batch gradient descent.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe geometric representation of a loss function over parameter space, where height indicates loss value for each parameter setting.

Compare how 2 books explain loss surface →

Related concepts

• gradient descentused by · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• momentumused by · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
• parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• hypersurfaceequivalent to · A surface of arbitrary dimension used here as the geometric shape of a loss surface in high-dimensional space.
• mean squared errorequivalent to · Mean squared error is the arithmetic mean of the squared differences between corresponding predicted and target numerical values.
• objective functionequivalent to · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.