A pretrained model has weights fitted on a source dataset or task before reuse. Transfer learning starts from those weights and adapts the model to a new problem, often replacing its final layer. This reduces training time and labeled-data requirements when source and target data share useful structure. section 13.0 section 13.6
Choose how much of the pretrained model can change:
- Freeze the original weights by disabling gradients, then train a newly fitted output layer. This is effective with very little labeled data but limits later improvement. section 13.4.2 section 13.3
- Use a warm start by allowing the reused weights to update. Warm models can surpass frozen models when the target dataset provides more labeled examples. section 13.3
- Mix both strategies by freezing early, general-purpose layers and updating deeper, task-specific layers; selecting the boundary requires experimentation. section 13.3
Small-image classification with ResNet18
Instantiate ResNet18 with pretrained weights, replace its final layer with a two-class linear layer, normalize inputs, and compare regular, warm-start, and frozen training on a tiny dataset. section 13.3
model = torchvision.models.resnet18() #1
model.fc = nn.Linear(model.fc.in_features, 2)
normal_small_results = train_network(model, loss, train_loader_small,
epochs=10, device=device, test_loader=test_loader,
score_funcs={’Accuracy’: accuracy_score})
model = torchvision.models.resnet18(pretrained=True) #2
model.fc = nn.Linear(model.fc.in_features, 2) #3
model = NormalizeInput(model)
warmstart_small_results = train_network(model, loss, train_loader_small,
epochs=10, device=device, test_loader=test_loader,
score_funcs={’Accuracy’: accuracy_score})
model = torchvision.models.resnet18(pretrained=True) #4
for param in model.parameters(): #5
param.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 2) #6
model = NormalizeInput(model)
frozen_transfer_small_results = train_network(model, loss,
train_loader_small, epochs=10, device=device, test_loader=test_loader,
score_funcs={’Accuracy’: accuracy_score})
#1 1. Training from scratch
#2 2. Training the warm model
#3 Performs some surgery
#4 3. Training with frozen weights
#5 Turns off gradient updating for all parameters
#6 Our new fc layer has requires_grad = True by default.
Inspect the PyTorch implementation that contrasts training from scratch, warm-start reuse, and frozen transfer with ResNet18. section 13.3
class BertBasedClassifier(nn.Module): #1
def __init__(self, bert_model, classes):
"""
bert_model: the BERT-based classification model to use as a frozen
initial layer of the network
classes: the number of output neurons/target classes for this
classifier.
"""
super(BertBasedClassifier, self).__init__()
self.bert_model = bert_model #2
self.attn = AttentionAvg( #3
AdditiveAttentionScore(
bert_model.config.dim))
self.fc1 = nn.Linear(bert_model.config.dim, #4
bert_model.config.dim)
self.pred = nn.Linear(bert_model.config.dim, classes) #5
def forward(self, input):
mask = getMaskByFill(input) #6
with torch.no_grad(): #7
x = self.bert_model(input)[0] #8
cntxt = x.sum(dim=1)/(mask.sum(dim=1).unsqueeze(1)+1e-5) #9
x = self.attn(x, cntxt, mask) #10
x = F.relu(self.fc1(x)) #11
return self.pred(x)
bertClassifier = BertBasedClassifier(bert_model, NUN_CLASS) #12
bert_results = train_network(bertClassifier, nn.CrossEntropyLoss(),
train_text_bert_loader, test_loader=test_text_bert_loader,
device=device, epochs=10, score_funcs={’Accuracy’: accuracy_score})
#1 Our new class for frozen training of BERT models
#2 We get a tensor of shape (B, T, D) from BERT, so we define a few of our own layers to get from (B, T, D) to a prediction of shape (B, classes)
#3 Attention to get down to (B, D) shape
#4 Does a little feature extraction
#5 Makes a prediction about the class
#6 Input is (B, T).
#7 This with no_grad() does the freezing.
#8 Hugging Face returns a tuple, so unpack it! (B, T, D)
#9 Computes the average embedding
#10 Applies the attention
#11 Makes predictions and returns
#12 Builds the classifier
Inspect the frozen BERT classifier structure, including no-gradient execution for the pretrained model and trainable classification layers. section 13.4.2
