Loss surface
A loss surface is a function that assigns a loss value to each possible setting of a model’s parameters.
Math and Architectures of Deep Learning
The geometric representation of a loss function over parameter space, where height indicates loss value for each parameter setting.
- How it is used
- Frames loss minimization as navigation across parameter space during supervised training.
- How deep it goes
- Defines the surface, maps it to geometry, and links it to gradients and minimization, but stops short of algorithmic variants beyond basic descent.
- Examples
- Uses concentric circles, concentric spheres, coaxial cylinders, heat maps, and a canyon or ravine analogy.
- What you can do after
- Apply gradient descent and momentum to move toward lower loss on a neural network surface.
Read how Math and Architectures of Deep Learning explains loss surface →
Sutskever's List
A loss landscape is the function relating a model's parameter values to its loss across parameter space.
- How it is used
- Frames the apparent difficulty of training deep networks and explains how AlexNet-era optimization challenged that view.
- How deep it goes
- Explains terrain metaphors, valley curvature, high-dimensional behavior, and mini-batch smoothing without formal analysis.
- Examples
- Uses a long, narrow valley to contrast oscillating large steps with slow progress along the valley’s base.
- What you can do after
- Understand why high-dimensional loss surfaces and mini-batch gradient averaging complicate the expectation that optimization gets trapped in poor minima.
Related concepts
- gradient descentused by · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
- momentumused by · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
- parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
- hypersurfaceequivalent to · A surface of arbitrary dimension used here as the geometric shape of a loss surface in high-dimensional space.
- mean squared errorequivalent to · Mean squared error is the arithmetic mean of the squared differences between corresponding predicted and target numerical values.
- objective functionequivalent to · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.