liveBook · Manning

1F1B schedule

As explained in Build a DeepSeek Model (From Scratch)

A 1F1B schedule is a pipeline-parallel execution schedule that interleaves one forward pass with one backward pass across successive micro-batches.

The 1F1B schedule is a pipeline-parallel execution strategy that interleaves one forward pass with one backward pass across successive micro-batches. It replaces GPipe’s run-all-forwards-then-run-all-backwards pattern with earlier gradient work, reducing GPU idle time. section 6.5.4 section 6.5.8

A pipeline moves through three operating phases:

  • Warm-up fills the pipeline with initial forward passes, leaving a small unavoidable startup bubble. section 6.5.4
  • Steady state alternates forward and backward work: after forwarding micro-batch i, a stage processes an available gradient for an earlier micro-batch instead of waiting. section 6.5.4
  • Cool-down drains the remaining backward passes after new forward work ends; in a true schedule, these final backward passes run back-to-back. section 6.5.4

GPU 2 in the steady state

GPU 2 may forward micro-batches 1 and 2 during warm-up. Once the gradient for micro-batch 1 arrives from GPU 3, it alternates backward work for that micro-batch with forward work for later micro-batches, keeping its timeline densely occupied. section 6.5.4

Figure 6.7 The 1F1B interleaved schedule. Forward (blue) and backward (green) passes alternate during the steady state. Note: any small gaps visible in this rendered timeline are artifacts of the profiling resolution, not the schedule itself. In a true 1F1B, each GPU executes its remaining backward passes back-to-back during cool-down.

Use the timeline to inspect the alternating forward and backward blocks, the reduced bubble, and the cool-down behavior. section 6.5.4

Efficiency = (Number of Micro batches) / (Number of Microbatches + Number of Stages - 1)

Estimate ideal schedule efficiency as micro-batches divided by micro-batches plus pipeline stages minus one. section 6.5.4

1F1B reduces, but does not eliminate, the pipeline bubble and activation-memory cost. In mixture-of-experts models, required All-to-All communication can still limit efficiency, motivating folded schedules such as DualPipe. section 6.5.8 section 6.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains warm-up, steady-state alternation, cool-down, efficiency, and remaining memory and communication costs.
Examples
Uses a GPU 2 timeline, successive micro-batches, a schedule diagram, and an efficiency formula.
What it assumes
Assumes familiarity with micro-batches, forward and backward passes, pipeline stages, and pipeline parallelism.
What you can do after
Evaluate pipeline occupancy and identify when communication or memory costs still constrain training efficiency.
How it connects
Connects pipeline parallelism and GPipe with micro-batches, mixture-of-experts communication, Expert Parallelism, and DualPipe.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.