Loss surface
A loss surface is a function that maps model parameter values to their corresponding loss values.
Math and Architectures of Deep Learning
The geometric representation of a loss function over parameter space, where height indicates loss value for each parameter setting.
- How it is used
- Frames loss minimization as navigation across parameter space during supervised training.
- How deep it goes
- Defines the surface, maps it to geometry, and links it to gradients and minimization, but stops short of algorithmic variants beyond basic descent.
- Examples
- Uses concentric circles, concentric spheres, coaxial cylinders, heat maps, and a canyon or ravine analogy.
- What you can do after
- Apply gradient descent and momentum to move toward lower loss on a neural network surface.
Read how Math and Architectures of Deep Learning explains loss surface →
Sutskever's List
A loss landscape is the function relating a model's parameter values to its loss across parameter space.
- How it is used
- Frames the apparent difficulty of training deep networks and explains how AlexNet-era optimization challenged that view.
- How deep it goes
- Explains terrain metaphors, valley curvature, high-dimensional behavior, and mini-batch smoothing without formal analysis.
- Examples
- Uses a long, narrow valley to contrast oscillating large steps with slow progress along the valley’s base.
- What you can do after
- Understand why high-dimensional loss surfaces and mini-batch gradient averaging complicate the expectation that optimization gets trapped in poor minima.
Timeless Algorithms, The Foundational Ideas
A loss surface is a geometric representation of how a model's loss varies across possible parameter values.
- How it is used
- Frames neural-network training as searching parameter values for low loss while prioritizing performance on unseen observations.
- How deep it goes
- Explains non-convex geometry and optimization behavior, but offers no guarantee of reaching a global minimum.
- Examples
- Compares neural-network loss surfaces with convex SVM objectives and notes that training runs can differ with initialization and batch order.
- What you can do after
- Evaluate neural-network solutions by performance on unseen observations rather than training loss alone.
Read how Timeless Algorithms, The Foundational Ideas explains loss surface →
What to know first
- optimizationOptimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- activation functionAn activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
- interceptAn intercept is an additive constant parameter in a model that shifts its output independently of its input features.
Related concepts
- gradient descentused by · Gradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
- momentumused by · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
- parameteruses · A parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- hypersurfaceequivalent to · A surface of arbitrary dimension used here as the geometric shape of a loss surface in high-dimensional space.
- mean squared errorequivalent to · Mean squared error is the arithmetic mean of the squared differences between corresponding predicted and reference numerical values.
- objective functionequivalent to · An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
Questions about loss surface
What is loss surface?
A loss surface is a function that maps model parameter values to their corresponding loss values.
Which Manning books explain loss surface?
Math and Architectures of Deep Learning, Sutskever's List and Timeless Algorithms, The Foundational Ideas.
What should I know before learning loss surface?
Manning books that teach loss surface usually introduce optimization, parameter, activation function and intercept first.
What is loss surface also called?
Loss surface is also called loss landscape.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


