liveBook · Manning

Loss surface

As explained in Timeless Algorithms, The Foundational Ideas

A loss surface is a geometric representation of how a model's loss varies across possible parameter values.

A loss surface represents how a neural network’s loss changes across possible values of its weights and biases. Even for a modest network, it has too many dimensions to visualize directly. Neural-network loss surfaces are generally non-convex, with valleys, flat regions, saddle points, and potentially multiple local minima. section 11.6.1

Gradient descent follows the surface’s local slope, but is not guaranteed to reach the global minimum. Its result can depend on initial parameters, mini-batch order, optimization method, and learning rate. The deepest point in training loss is not necessarily the best choice: the practical aim is good performance on unseen observations. section 11.6.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains non-convex geometry and optimization behavior, but offers no guarantee of reaching a global minimum.
Examples
Compares neural-network loss surfaces with convex SVM objectives and notes that training runs can differ with initialization and batch order.
What you can do after
Evaluate neural-network solutions by performance on unseen observations rather than training loss alone.
How it connects
Connects neural-network training to gradient descent, non-convex optimization, support vector machines, and generalization.

Also explained in

Compare how 3 books explain loss surface →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.