A forward pass propagates inputs through a neural network to produce outputs. It is the primary prediction computation during inference; in training, it precedes the backward pass that calculates gradients. For a linear layer, the core operation is y = Wx. section 5.6.1
In autoregressive generation, a simple loop passes the entire sequence so far to the model, selects a next token from the final-position logits, and appends it. The next iteration passes the longer sequence through the model again. This makes the cost of a full computational pass visible for every generated token. section 2.1.3
Listing 2.1 Visualizing autoregressive generation with GPT-2
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
prompt = "The next day is bright"
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs.input_ids
print(f"Prompt: '{prompt}'", end="")
for _ in range(20):
outputs = model(input_ids) #A
logits = outputs.logits
next_token_logits = logits[:, -1, :] #B
next_token_id = next_token_logits.argmax(dim=-1).unsqueeze(-1)
input_ids = torch.cat([input_ids, next_token_id],
[CA] dim=-1) #C
new_token = tokenizer.decode(next_token_id[0])
print(new_token, end="", flush=True)
print("\n")
Trace model(input_ids) inside the loop, followed by final-position logit selection and token concatenation. section 2.1.3
The forward pass can also be a precision trade-off. In a mixed-precision linear layer, BF16 input activations and high-precision weights are converted to FP8 for multiplication. The result is accumulated in FP32 for stability, then converted to BF16 for storage. FP8 matrix multiplication requires compatible hardware. section 5.6.1
Figure 5.23 The data flow and precision formats for the forward pass.

Follow the formats used at each point in the linear layer's forward computation. section 5.6.1
How a forward pass is distributed affects utilization. In a GPipe fill phase, micro-batches move through pipeline stages in order; each GPU must wait for the preceding stage's activations. With four GPUs, later stages initially sit idle, creating part of the pipeline bubble. section 6.5.3
Figure 6.4 The forward pass in a naive pipeline schedule. Each GPU must wait for the previous one to complete its work on a micro-batch. The white area to the left of the bars represents the initial idle time (the "bubble") as the pipeline fills up.

Inspect the staggered forward passes and idle time as the pipeline fills. section 6.5.3
Data parallelism divides the batch differently: each GPU runs a forward pass on its own data slice, and the outputs are gathered on the main GPU. In nn.DataParallel, model replication on every forward pass and centralized gathering contribute to load imbalance and poor scalability. section 6.4.4













