Gradient norm summarizes the size of parameter-update signals. During full supervised fine-tuning, its logged trajectory is a quick check on optimization stability; judge the trend, not just the first value. section 6.2
Use the trajectory as a practical warning system:
- Norms that stabilize and stay bounded are expected. For full SFT at learning rate 2e-5 on a 4B model, they typically settle around 2–5 after the first steps. section 6.2
- Sustained norms above 10 or a rising trend suggest the learning rate is too high; a sudden spike into the hundreds may signal divergence. section 6.2 section 6.1.3
- Norms collapsing toward zero may mean training has reached a flat region or become stuck. A NaN norm signals instability; stop and adjust the run. section 6.2 section 6.1.3
A full-SFT run
In one run, the norm starts at 5.66 and settles around 3–5. The initial value exceeds the monitor’s conservative 5.0 warning threshold, but the subsequent stabilization makes it informational rather than proof of a failing run. section 6.1.3 section 6.2.2
Listing 6.4 Training monitor (chapter06/scripts/monitor.py)
def monitor(run_dir: str) -> None:
state_file = Path(run_dir) / "trainer_state.json"
state = json.loads(state_file.read_text())
history = state.get("log_history", [])
# Separate training and eval entries
train_entries = [e for e in history if "loss" in e and "eval_loss" not in e]
eval_entries = [e for e in history if "eval_loss" in e]
# ... (print first/latest training loss and change) ...
# Warn if overfitting #A
if len(eval_entries) >= 2:
recent = eval_entries[-1]["eval_loss"]
previous = eval_entries[-2]["eval_loss"]
if recent > previous + 0.02:
print(" WARNING: Validation loss increased"
" between last two evals.")
print(" This may indicate overfitting. Consider stopping early.")
# Gradient norm check #B
grad_entries = [e for e in history if "grad_norm" in e]
if grad_entries:
max_norm = max(e["grad_norm"] for e in grad_entries)
if max_norm > 5.0:
print(" WARNING: Gradient norm exceeded 5.0"
" -- consider reducing learning rate.")Inspect the training monitor for an example of checking logged gradient norms alongside training and validation loss. section 6.2.2
