A hyperparameter is a configuration value set before training that changes how a learning process behaves and affects the resulting model. Unlike learned parameters, it is chosen by the practitioner and may require iteration and measurement to optimize. section 13.4.3
Common training hyperparameters control the following parts of a model-training run:
- Epochs specify how many times training iterates over the dataset; too many can cause overfitting. section 14.3.2
- Batch sizes specify how many examples are processed together and are constrained by available GPU memory. section 14.3.2
- Warmup steps gradually adjust the learning rate at the start of training, while weight decay reduces overfitting by shrinking weights at each step. section 14.3.2
- The number of latent features in matrix factorization determines the dimensions used to represent users and items. section 9.3.1
Configuring a question-answering trainer
A Transformer training run can use three epochs, training and evaluation batch sizes of 16 and 64, 500 warmup steps, and 0.01 weight decay. Grid search evaluates combinations of such values to identify settings that improve model quality. section 14.3.2
Listing 14.10 Initializing the trainer and its hyperparameters
from transformers import RobertaForQuestionAnswering, TrainingArguments,
Trainer, default_data_collator
model = RobertaForQuestionAnswering.from_pretrained(
"deepset/roberta-base-squad2")
training_args = TrainingArguments(
evaluation_strategy="epoch", #1
num_train_epochs=3, #2
per_device_train_batch_size=16, #3
per_device_eval_batch_size=64, #4
warmup_steps=500, #5
weight_decay=0.01, #6
logging_dir="data/question-answering/logs",
output_dir="data/question-answering/results")
trainer = Trainer(
model=model, #7
args=training_args, #8
data_collator=default_data_collator,
tokenizer=tokenizer,
train_dataset=tokenized_datasets["train"], #9
eval_dataset=tokenized_datasets["test"]) #10
Inspect the executable TrainingArguments configuration and see how hyperparameters connect the pretrained model, datasets, evaluation schedule, logging, and output directories. section 14.3.2
