liveBook · Manning

Batching

As explained in Build a DeepSeek Model (From Scratch)

A micro-batch is a smaller subdivision of a training batch that can be processed as an individual computation unit.

Batching groups training examples for a weight update. A mini-batch gives a less noisy gradient than a single example, but the whole mini-batch must fit in GPU memory. A micro-batch is a smaller chunk of a batch that can be processed as a separate computation unit. section 6.4.1 section 6.5.1

In a standard training step, the model computes predictions for a mini-batch, averages the loss, and backpropagates to calculate gradients. The optimizer updates the weights, then clears the gradients before the next mini-batch. For a batch of B examples, the batch gradient is the average of their individual gradients. section 6.4.1

g_batch = (1/B) * (g₁ + g₂ + ... + g_B)

The batch gradient averages the gradients of the B examples. section 6.4.1

Splitting batches supports different scaling strategies:

  • Gradient accumulation processes smaller mini-batches sequentially and accumulates their gradients before one weight update. This simulates a larger effective batch on memory-constrained hardware. section 6.6
  • Data parallelism assigns batch shards to GPUs with identical model replicas. Each GPU computes local gradients; the gradients are then aggregated and averaged so replicas apply the same update. section 6.4.3
  • Pipeline parallelism splits the model into stages and feeds micro-batches through them. Different stages can work on different micro-batches, but stages must wait for inputs or gradients from their neighbors. section 6.5.1 section 6.5.3

Filling a four-GPU pipeline

A batch of 32 can be split into eight micro-batches of four. In a four-stage pipeline, the first GPU starts the first micro-batch; the next GPU can start it only after receiving its activations. Meanwhile, the first GPU can start the second micro-batch. The initial waiting time forms part of the pipeline bubble. section 6.5.1 section 6.5.3

Figure 6.4 The forward pass in a naive pipeline schedule. Each GPU must wait for the previous one to complete its work on a micro-batch. The white area to the left of the bars represents the initial idle time (the "bubble") as the pipeline fills up.

The forward-pass timeline shows how dependencies stagger the stages as the pipeline fills. section 6.5.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the training-step sequence, batch splitting and gradient synchronization, then traces pipeline filling and its idle time.
Examples
Traces eight micro-batches of four through a four-GPU pipeline and describes shards processed by replicated models.
What you can do after
Evaluate how batch splitting addresses memory constraints and exposes parallel work, while recognizing pipeline idle time.
How it connects
Connects batching to gradient accumulation, data parallelism, pipeline parallelism and the pipeline bubble.

Also explained in

Compare how 23 books explain batching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.