liveBook · Manning

Backpropagation

As explained in LLMs in Production

Backpropagation is the procedure for computing parameter gradients by propagating error derivatives backward through a neural network.

Backpropagation turns prediction error into updates for a neural network’s weights. During language-model training, predictions are compared with target tokens, and the resulting loss guides parameter adjustments alongside Adam or stochastic gradient descent. section 5.2.1

Predicting the next token

A model receives tokenized text and predicts the next word in each sequence. The difference between its predictions and the actual words produces the loss; backpropagation uses that error to adjust the model’s internal parameters. Repeating this process improves predictive accuracy, although large-scale training can take weeks or months. section 5.2.1

Figure 5.2 A simplified version of all the steps necessary to train a language model (large or otherwise) from scratch. You must have data, then define all of the model behavior, and only then proceed to train.
figure

Trace where backpropagation fits in the larger from-scratch workflow: prepare data, define model behavior, then train and validate the model. section 5.2.1

How LLMs in Production covers it

How deep it goes
Explains its role in minimizing loss but stops short of derivative calculations or implementation internals.
Examples
Uses next-word prediction, tokenized text, and a small from-scratch training example.
What it assumes
Assumes a model, dataset, learning objective, and tokenized numerical inputs already exist.
What you can do after
Practice a miniature from-scratch language-model training workflow using the provided code.
How it connects
Connects autoregressive learning, loss, optimization, validation datasets, and fine-tuning.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 17 books explain backpropagation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.