A hyperparameter is a training or model setting chosen before fitting. It shapes how a model is configured or trained rather than representing a learned weight. Examples include learning rate, epoch count, sequence length, embedding dimension, decoder-block count, attention-head count, vocabulary size, block size, dropout, and batch size. section 2.2.1 section 11.2.3
Hyperparameters matter because they affect model quality, training speed, resource use, and overfitting. Validation data helps compare settings such as learning rate and training duration without selecting them directly from the training data. section 2.2.1 section 12.5
Common uses include:
- Choose the number of epochs and use validation loss to determine when training should stop; an EarlyStop setting such as patience=10 waits for improvement before stopping. section 2.4.1
- Configure a GPT by setting sequence length, embedding dimension, decoder-block count, dropout rates, vocabulary size, and attention-head count. section 12.1 section 11.2.3
- Control a MuseGAN training loop with batch size 64, five critic updates per iteration, output every 10 epochs, and 500 total epochs. section 13.5.1
Scaling a GPT configuration
A GPT-2XL-style configuration uses 48 decoder layers, 25 attention heads, 1,600-dimensional token embeddings, a 50,257-token vocabulary, a 1,024-token block size, and 0.1 dropout rates. Increasing settings may improve learning capacity but can lengthen training and encourage overfitting; reducing them may produce incoherent text. section 11.2.3 section 12.5
Listing 11.3 Specifying hyperparameters in GPT-2XL
class Config(): #1
def __init__(self):
self.n_layer = 48
self.n_head = 25
self.n_embd = 1600
self.vocab_size = 50257
self.block_size = 1024
self.embd_pdrop = 0.1
self.resid_pdrop = 0.1
self.attn_pdrop = 0.1 #2
config=Config() #3
The configuration listing makes the relationship between named hyperparameters and GPT architecture directly inspectable. section 11.2.3
