Batch size is the amount of training data processed together before the model updates its weights. In Transformer language-model training, that amount is often described as a fixed number of tokens, such as 32. Batching turns a long token stream into manageable pieces that fit within the model’s context window. section 11.10 section 11.32
A batch moves through training as a short, repeatable cycle:
- Select the next batch from the dataset. section 11.32
- Run the batch through the model, calculate its loss, and backpropagate the resulting gradients. section 11.32
- Update the model’s weights, then repeat with another batch. section 11.32
- Handle an incomplete final batch by dropping it, padding it, or using its variable length when the architecture allows. section 11.13
A 32-token batch
Suppose a token stream is divided using a context window of 32. The first batch contains 32 token IDs and passes through the Transformer as one input. After its loss produces gradients and the weights are updated, training selects the next batch. If the final batch contains only three tokens, it can be dropped, padded with special tokens that the model ignores, or processed directly by an architecture supporting variable-length input. section 11.13 section 11.32
Figure 11.20 Tokenized training data as training batches
Visualizes how a tokenized stream becomes separate training batches. section 11.10
def batch_tokens(token_stream, context_window_size):
batches = []
start_index = 0
while True:
end_index = start_index + context_window_size #A
if end_index > len(token_stream): #B
break
batch = token_stream[start_index:end_index] #C
batches.append(batch)
start_index += 1 #D
return batches
Shows a Python-style implementation that slices a token stream into fixed-length batches. section 11.10
