A model parameter is a numerical value that a neural network adjusts during training. Weights are parameters: their learned values help determine the network’s predictions. An optimizer updates them using information from the loss gradient. A learning rate controls how much the weights change in an update; unlike the weights, it is a hyperparameter selected for training. section 2.2.1
From image predictions to trained weights
In an image classifier, a forward pass produces predictions, and cross-entropy loss compares them with the image labels. Adam then updates the network’s weights using the loss gradient. Other models update parameters for different goals: a variational autoencoder adjusts its encoder and decoder to reduce reconstruction loss and KL divergence, while an anime-face generator adjusts its parameters as a discriminator learns to identify generated images. section 2.2.1 section 7.3.2 section 1.2.2
Architecture and learned values are different things. A newly constructed GPT-2XL has the model structure but cannot yet generate meaningful text: its pretrained weights must be loaded. Training a model with that many parameters is impractical without substantial computing facilities and data. Once loaded, the weights support repeated next-token predictions from a prompt. section 11.3 section 11.3.2
A weight matrix must fit its destination
A GPT-2XL feed-forward weight in a Hugging Face Conv1D layer has shape `(1600, 6400)`. The corresponding weight in a PyTorch linear layer has shape `(6400, 1600)`. Copying the values therefore requires a transpose. Other matching parameters can be copied without one; `attn.masked_bias` is excluded because the destination model already supplies its own masking buffer. section 11.3.1
transposed = ['attn.c_attn.weight', 'attn.c_proj.weight',
'mlp.c_fc.weight', 'mlp.c_proj.weight'] #1
for k in keys:
if any(k.endswith(w) for w in transposed):
with torch.no_grad():
sd[k].copy_(sd_hf[k].t()) #2
else:
with torch.no_grad():
sd[k].copy_(sd_hf[k]) #3
The transfer loop selects weight names that need transposing, then copies each pretrained tensor into the corresponding model parameter. section 11.3.1
Parameter count also affects model design. Convolutional layers reuse weights across image regions and need fewer parameters than similarly sized fully connected networks, making them useful for high-resolution images. Whether parameters are learned through training or loaded from a pretrained model, their values—and their fit with the architecture—matter. section 4.2 section 11.3.1
