liveBook · Manning

Objective function

As explained in CUDA for LLMs

Loss is a numerical measure of the discrepancy or error that a model's training process seeks to minimize.

A loss is a scalar measure of the discrepancy between a network’s output and the true labels. Training seeks to minimize it. section 4.2.2

For a linear layer, the loss gradients are computed as:

How CUDA for LLMs covers it

How deep it goes
Defines loss and gives linear-layer gradient formulas, but does not specify a loss function.
Examples
Uses a linear layer to show weight, input, and bias gradient calculations.
What you can do after
Apply matrix and reduction formulas to calculate linear-layer gradients from the loss.
How it connects
Links loss gradients to linear layers, matrix multiplication, and batch-wise gradient accumulation.

Also explained in

Compare how 42 books explain objective function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of CUDA for LLMs. The section links point to the parts of the book it draws on.