Batch size is the number of examples processed together in a training or evaluation step. Instead of updating a neural network from one example’s error, a training step can use the mean error across a batch. The choice affects both how training proceeds and how much data must fit in memory at once. section 5.3 section 9.3.4
For a word-level RNN, reading tokens one at a time can be slow. Batchifying text lets the model work on multiple passages simultaneously: passages become rows or columns of a tensor that PyTorch can process efficiently on a GPU. A small experiment splits a stream of word IDs into groups of eight, then stacks the groups into one tensor. section 8.4.1
When choosing a batch size, check more than whether the first step runs:
- Check sequence length and GPU memory together. Increasing a BERT classifier’s maximum sequence length from its default of 128 to 300 may require smaller training and evaluation batches. section 9.3.4
- Watch for out-of-memory errors during both training and evaluation; they may appear late in a run. The largest batch that fits is not necessarily the best choice: smaller batches can make gradient descent more stochastic. section 9.3.4
- For GPT-2 fine-tuning, start with a single-digit batch size and adjust for available GPU RAM. A data collator builds batches from dataset samples and handles preprocessing such as padding. section 10.2.2
Listing 9.38 Setting up parameters for evaluation during training and early stopping
>>> model_type = 'bert'
>>> model_name = 'bert-base-cased'
>>> output_dir = f'{model_type}-example2-outputs' #1
>>> best_model_dir = f'{output_dir}/best_model'
>>> model_args = {
... 'output_dir': output_dir,
... 'overwrite_output_dir': True,
... 'manual_seed': random_state,
... 'no_cache': True,
... 'best_model_dir': best_model_dir,
... 'max_seq_length': 300,
... 'train_batch_size': 24,
... 'eval_batch_size': 24,
... 'gradient_accumulation_steps': 1,
... 'learning_rate': 5e-5,
... 'evaluate_during_training': True,
... 'evaluate_during_training_steps': 1000,
... 'save_eval_checkpoints': False,
... "save_model_every_epoch": False,
... 'save_steps': -1, # saving model unnecessarily takes time during training
... 'reprocess_input_data': True,
... 'num_train_epochs': 5, #2
... 'use_early_stopping': True,
... 'early_stopping_patience': 4, #3
... 'early_stopping_delta': 0,
... }
Inspect the BERT configuration: max_seq_length is 300, while train_batch_size and eval_batch_size are each set to 24. section 9.3.4
Listing 10.11 Defining training arguments for GPT-2 fine-tuning
>>> from nlpia2.constants import DATA_DIR #1 >>> from transformers import TrainingArguments >>> from transformers import DataCollatorForLanguageModeling >>> training_args = TrainingArguments( ... output_dir=DATA_DIR / 'ch10_checkpoints', ... per_device_train_batch_size=5, ... num_train_epochs=5, ... save_strategy='epoch') >>> collator = DataCollatorForLanguageModeling( ... tokenizer=tokenizer, mlm=False) #2
Compare the GPT-2 setup, which sets per_device_train_batch_size to 5 in TrainingArguments and supplies a language-modeling data collator. section 10.2.2
Treat batch size as a practical setting to test alongside sequence length and hardware capacity, not a value to maximize automatically. section 9.3.4 section 10.2.2
