Training loop
A training loop is an iterative procedure that processes training data, evaluates an objective, and updates model parameters.
Deep Learning with R, Third Edition
A training loop is a repeated process that evaluates examples, adjusts model parameters, and progressively minimizes the loss function.
- How it is used
- Trains models by repeatedly evaluating batches and adjusting weights to reduce loss.
- How deep it goes
- Explains batch steps and epochs, then implements both a simplified full loop and a stateless gradient update.
- Examples
- Uses MNIST handwritten-digit classification and a JAX linear classifier to illustrate built-in and custom loops.
- What you can do after
- Apply it via Keras fit(), or build a custom training loop from scratch for more control.
Read how Deep Learning with R, Third Edition explains training loop →
Rearchitecting LLMs
A training loop is an iterative procedure that computes model outputs, evaluates an error, and updates trainable parameters over batches and iterations.
- How it is used
- Recovers a pruned student model’s capabilities by iteratively aligning its behavior with a teacher.
- How deep it goes
- Walks through batching, forward passes, loss calculation, backpropagation, and optimizer updates, then compares recovery strategies.
- Examples
- Compares labels-only and feature-based distillation against teacher and pruned baselines using perplexity and capability retention at multiple data scales.
- What you can do after
- Apply it via PyTorch to update a student model from teacher outputs using logits-only or compound distillation losses.
LLMs in Production
A training loop is the repeated process of computing model outputs, evaluating loss, calculating gradients, and updating parameters.
- How it is used
- Organizes model training from data preparation through repeated prediction, loss, backpropagation, and parameter updates.
- How deep it goes
- Explains the loop from batches to gradients and implementation components, then extends into speed-memory tradeoffs without deriving optimization theory.
- Examples
- Uses a generative CBoW model, Hamlet tokens, a GPT built from scratch, and a speed-memory comparison table.
- What you can do after
- Build it from scratch with modular model components, batched data, loss calculation, backpropagation, and resource-aware training choices.
Build a DeepSeek Model (From Scratch)
A training loop is an iterative procedure that processes batches, computes predictions and loss, backpropagates gradients, updates parameters, and clears gradients.
- How it is used
- Runs iterative optimization that teaches models to reduce loss on batches and assesses generalization on validation data.
- How deep it goes
- Explains the complete update sequence, evaluation, scheduling, and checkpointing, but stops short of distributed training.
- Examples
- Uses a PyTorch language-model loop with TinyStories batches, validation loss, mixed precision, and best or periodic checkpoint files.
- What you can do after
- Build it from scratch by combining batches, model calls, loss computation, backpropagation, optimizer updates, evaluation, and checkpointing.
Read how Build a DeepSeek Model (From Scratch) explains training loop →
Fight Fraud with Machine Learning
A model training loop is an iterative procedure that updates a model using training data and evaluates its learning progress across repeated steps or epochs.
- How it is used
- Deploys iterative parameter updates and validation checks when training segmentation, audio, tabular, or graph-based fraud-detection models.
- How deep it goes
- Explains executable training and validation steps, metrics, and learning curves, but defers model architecture and broader test interpretation.
- Examples
- Uses receipt-image segmentation, transformer-based voice-faking detection, MLP transaction classification, and GCN modeling on the Elliptic Bitcoin dataset.
- What you can do after
- Build it from scratch in PyTorch, monitor training and validation metrics, and evaluate trained models on unseen data.
Read how Fight Fraud with Machine Learning explains training loop →
CUDA for LLMs
A training loop repeatedly processes training data, computes model outputs and loss, and updates model parameters.
- How it is used
- Establishes a verified training baseline and exposes step-level costs for comparing implementations.
- How deep it goes
- Traces batches through data loading, forward, loss, backward, and updates; flags CUDA synchronization and allocation costs without detailing optimized kernels.
- Examples
- Uses a 784–256–10 MLP with batches of eight and reports ten-epoch PyTorch and C timing results.
- What you can do after
- Diagnose training bottlenecks with stage-level timing and compare implementation correctness through epoch loss.
Graph Neural Networks in Action
A training loop is an iterative procedure that processes data through a model and updates its parameters across training iterations.
- How it is used
- Trains graph and spatiotemporal models while monitoring validation behavior and comparing generalization.
- How deep it goes
- Explains initialization, forward and backward passes, validation, scheduling, and testing, without deriving optimization mathematics.
- Examples
- Uses parallel GCN and GraphSAGE classification plus NRI trajectory prediction with scheduled learning rates and future-horizon MSE.
- What you can do after
- Apply it via PyTorch to train models, track losses, tune schedules, and evaluate predictions on held-out data.
Read how Graph Neural Networks in Action explains training loop →
Also covered in
What to know first
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- forward passA forward pass is a computation that propagates inputs through a model or dependency structure in the forward direction to calculate outputs or intermediate values.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- activation functionAn activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
Related concepts
- benchmarkproduces · A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.
- epochuses · An epoch is one complete pass through the training data during model training.
- fit methodimplemented by · A fit method is an operation that estimates model parameters or learns a model or data transformation from supplied data.
- kernel launchuses · A kernel launch is an operation that submits a GPU kernel for execution by GPU threads under a specified execution configuration.
Questions about training loop
What is training loop?
A training loop is an iterative procedure that processes training data, evaluates an objective, and updates model parameters.
Which Manning books explain training loop?
Deep Learning with R, Third Edition, Rearchitecting LLMs, LLMs in Production, Build a DeepSeek Model (From Scratch), Fight Fraud with Machine Learning, CUDA for LLMs and Graph Neural Networks in Action.
What should I know before learning training loop?
Manning books that teach training loop usually introduce objective function, forward pass, parameter and activation function first.
What are the parts of training loop?
Training loop includes gradient update.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






