liveBook · Manning

Learning rate

As explained in Building Specialized AI Systems

A learning rate is the optimization step-size parameter that controls how much model weights change during training.

The learning rate is the optimization step size: it controls how far model weights move on each update while training. In gradient descent, the update moves opposite the loss gradient; the learning rate determines the size of that downhill move. It is therefore a central control for both convergence speed and training stability. section 3.1.2

Choosing a rate is a trade-off. A rate that is too large can overshoot lower-loss settings, make loss fluctuate, and prevent convergence. A rate that is too small produces stable but painfully slow progress. For LLMs, cited values range from 1e-5 to 1e-3; pre-training commonly uses 1e-4 to 3e-4, while fine-tuning commonly uses 1e-5 to 5e-5. section 3.1.2

During fine-tuning, set the learning rate alongside related controls:

  • Use a lower rate than pre-training when preserving the foundation model’s learned representations matters; fine-tuning should make careful adaptations rather than dramatic changes. section 1.4.2
  • Balance it with batch size and gradient accumulation, which affect update stability, efficiency, and resource needs. section 1.4.2
  • Manage epochs and use evaluation or early stopping to avoid overfitting training examples instead of learning generalizable patterns. section 1.4.2

Analyst-report summarization configuration

A fine-tuning run formats each analyst report as an input and its verified summary as the target, then configures training with a learning rate of 2e-4, batch size 2, gradient accumulation of 4, warmup steps, a linear scheduler, logging, and evaluation. Together, these settings govern how quickly weights move toward producing useful summaries without simply memorizing report phrasing. section 2.5.3 section 2.5.3

Figure 3.5 Gradient descent principle illustrated with a single weight variable. For LLMs, the same principle applies to billions of parameters.

Use the single-weight landscape illustration to connect a numerical learning rate to the practical choice between overshooting, slow progress, and steady descent. section 3.1.2

How Building Specialized AI Systems covers it

How deep it goes
Explains the gradient-descent step, numerical ranges, and too-large versus too-small failure modes without deriving gradients.
Examples
Configures 2e-4 for LoRA-based analyst-report summarization with batch size, accumulation, warmup, and a linear scheduler.
What it assumes
Introduces gradient descent through a downhill-landscape analogy while naming gradients, loss, parameters, and weight updates.
What you can do after
Operate/configure fine-tuning by selecting a rate that balances adaptation speed, stability, and preservation of learned representations.
How it connects
Connects learning rate with gradient descent, loss, batch size, gradient accumulation, epochs, early stopping, and overfitting.

Also explained in

Compare how 23 books explain learning rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.