Hyperparameter tuning searches for settings chosen before training rather than learned as model parameters. Those settings can change both training efficiency and final performance. For example, a learning rate of 0.1 may converge faster than 0.01, while 0.01 may converge more steadily and yield greater precision. Because training large language models is computationally intensive, the aim is to compare useful settings without wasting runs. section 8.4 section 8.1.1
A practical tuning loop has four parts:
- Choose a search space, such as candidate learning rates, epoch counts, batch sizes, and weight-decay values. Grid search, random search, and Bayesian optimization are available strategies. section 8.1.1 section 8.2
- Train and evaluate each candidate against a named metric. In a Ray Tune setup, a TorchTrainer receives training and validation datasets, while the Tuner minimizes evaluation loss. section 8.2
- Control the cost of the search. The Ray configuration uses an ASHA scheduler; a W&B sweep configuration shows Hyperband early termination. section 8.2 section 8.2.1
- Inspect results, select the best configuration, and train with it. Log run settings and metrics so differences between candidates remain visible. section 8.2 section 8.2.1
Sequence-classification tuning
For the SuperGLUE CommitmentBank task, convert Hugging Face train, validation, and test splits to Ray datasets. Tokenize each premise–hypothesis pair and prepare its classification label. The illustrated search varies learning rate, epochs, batch size, and weight decay, selects the trial with the lowest evaluation loss, then passes that trial’s configuration to a new TorchTrainer fit. The example connects the search space to a complete training workflow rather than treating candidate values as isolated guesses. section 8.2
Listing 8.5 Instantiating the Tuner class
tuner = Tuner(
trainer,
param_space=\{
"train_loop_config": \{
"learning_rate": tune.grid_search([2e-5, 2e-4, 2e-3, 2e-2]),
"epochs": tune.choice([2, 4, 6, 8]),
"batch_size": tune.choice([16, 32, 64, 128]),
"weight_decay": tune.grid_search([0.0, 0.01, 0.1, 0.001])
\}
\},
tune_config=tune.TuneConfig(
metric="eval_loss",
mode="min",
num_samples=1,
scheduler=ASHAScheduler(
max_t=max([2, 4, 6, 8]),
grace_period=1,
reduction_factor=2,
),
),
run_config=RunConfig(
name="tune_transformers",
checkpoint_config=CheckpointConfig(
num_to_keep=1,
checkpoint_score_attribute="eval_loss",
checkpoint_score_order="min",
),
),
)
The Tuner listing makes the search concrete: grid-searched learning rates and weight decay sit alongside sampled epoch and batch-size choices, with evaluation loss as the minimization target. section 8.2
Experiment tracking adds another way to run and judge searches. A W&B sweep specifies a search method, parameter ranges or choices, and a metric to optimize; its example uses Bayesian search to minimize evaluation loss. W&B records each run’s hyperparameters and training or validation metrics, then provides views for comparing combinations. GPU-usage monitoring can also inform whether a less costly device may suffice. section 8.2.1
Listing 8.9 Defining the sweep configuration
sweep_config = {
'method': 'bayes',
'metric': {
'name': 'eval_loss',
'goal': 'minimize'},
'parameters': {
'learning_rate': {
'min': 1e-5,
'max': 5e-4},
'num_train_epochs': {
'values': [2, 3, 4]},
'per_device_train_batch_size': {
'values': [8, 16, 32]},
'model_name_or_path': {
'values': ['bert-base-uncased', 'distilbert-base-uncased']}
},
'early_terminate': {
'type': 'hyperband',
'min_iter': 3,
'eta': 2,
's': 2 }
}
The sweep configuration shows how to declare Bayesian search, evaluation-loss minimization, candidate model and training settings, and early termination in one artifact. section 8.2.1
