liveBook · Manning

Gradient norm

As explained in LLM Customization and Fine-Tuning

The magnitude of a model’s gradient vector, which summarizes the size of parameter-update signals during training.

Gradient norm summarizes the size of parameter-update signals. During full supervised fine-tuning, its logged trajectory is a quick check on optimization stability; judge the trend, not just the first value. section 6.2

Use the trajectory as a practical warning system:

  • Norms that stabilize and stay bounded are expected. For full SFT at learning rate 2e-5 on a 4B model, they typically settle around 2–5 after the first steps. section 6.2
  • Sustained norms above 10 or a rising trend suggest the learning rate is too high; a sudden spike into the hundreds may signal divergence. section 6.2 section 6.1.3
  • Norms collapsing toward zero may mean training has reached a flat region or become stuck. A NaN norm signals instability; stop and adjust the run. section 6.2 section 6.1.3

A full-SFT run

In one run, the norm starts at 5.66 and settles around 3–5. The initial value exceeds the monitor’s conservative 5.0 warning threshold, but the subsequent stabilization makes it informational rather than proof of a failing run. section 6.1.3 section 6.2.2

Listing 6.4 Training monitor (chapter06/scripts/monitor.py)
def monitor(run_dir: str) -> None:
    state_file = Path(run_dir) / "trainer_state.json"
    state = json.loads(state_file.read_text())
    history = state.get("log_history", [])
 
    # Separate training and eval entries
    train_entries = [e for e in history if "loss" in e and "eval_loss" not in e]
    eval_entries = [e for e in history if "eval_loss" in e]
 
    # ... (print first/latest training loss and change) ...
 
    # Warn if overfitting                                                #A
    if len(eval_entries) >= 2:
        recent = eval_entries[-1]["eval_loss"]
        previous = eval_entries[-2]["eval_loss"]
        if recent > previous + 0.02:
            print("  WARNING: Validation loss increased"
                  " between last two evals.")
            print("  This may indicate overfitting. Consider stopping early.")
 
    # Gradient norm check                                                #B
    grad_entries = [e for e in history if "grad_norm" in e]
    if grad_entries:
        max_norm = max(e["grad_norm"] for e in grad_entries)
        if max_norm > 5.0:
            print("  WARNING: Gradient norm exceeded 5.0"
                  " -- consider reducing learning rate.")

Inspect the training monitor for an example of checking logged gradient norms alongside training and validation loss. section 6.2.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Interprets logged trajectories and warning thresholds without deriving gradient-norm mathematics.
Examples
Uses a Qwen3-4B full-SFT run whose norm starts at 5.66 and settles around 3–5.
What you can do after
Diagnose optimization instability by monitoring norm trends and investigating spikes, collapse, or NaN values.
How it connects
Connects gradient-norm trends with learning rate, training loss, validation loss, and optimization stability.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.