Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning method: it learns low-rank updates while keeping most pretrained weights fixed. By updating only a small fraction of parameters, it can achieve competitive performance on classification tasks. section 12.3.5
For protein classification, a pretrained ESM-2 model can produce an embedding for each sequence position. Mean pooling combines these embeddings into a sequence representation, which a classification head uses to predict a label. LoRA offers a way to adapt the pretrained model without training its entire backbone. section 12.3.5
Listing 12.6 Fine-tuning ESM-2 with a classification head (optional, hands-on)
device = "cuda" if torch.cuda.is_available() else "cpu"
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
backbone = AutoModel.from_pretrained(model_name)
class ESM2Classifier(nn.Module):
def __init__(self, backbone, num_classes, freeze_backbone=True):
super().__init__()
self.backbone = backbone
hidden = backbone.config.hidden_size
if freeze_backbone:
for p in self.backbone.parameters():
p.requires_grad = False
self.head = nn.Linear(hidden, num_classes)
def mean_pool(self, hidden_states, attention_mask):
mask = attention_mask.unsqueeze(-1).float() # (B, L, 1)
summed = (hidden_states * mask).sum(dim=1)
counts = mask.sum(dim=1).clamp(min=1e-9)
return summed / counts
def forward(self, input_ids, attention_mask):
out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
pooled = self.mean_pool(out.last_hidden_state, attention_mask)
return self.head(pooled) # (B, num_classes)
model = ESM2Classifier(backbone, num_classes=2, freeze_backbone=True).to(device)
optimizer = torch.optim.Adam(
(p for p in model.parameters() if p.requires_grad), lr=1e-4)
loss_fn = nn.CrossEntropyLoss()
def run_epoch(loader, train=True): # Loader yields (sequences, labels)
model.train() if train else model.eval()
total = 0.0
with torch.set_grad_enabled(train):
for sequences, labels in loader:
enc = tokenizer(list(sequences), padding=True, return_tensors="pt")
input_ids = enc["input_ids"].to(device)
attention_mask = enc["attention_mask"].to(device)
labels = torch.tensor(labels, dtype=torch.long, device=device)
logits = model(input_ids, attention_mask)
loss = loss_fn(logits, labels)
if train:
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
total += loss.item() * input_ids.size(0)
return total / len(loader.dataset)This classifier illustrates two alternatives to LoRA: training only the classification head or fine-tuning the whole backbone. It does not implement LoRA. section 12.3.5
For antibody classification, keep homologous sequences out of both the training and test sets. Cluster by sequence identity, or by clonotype and donor, and assign whole clusters to one side of the split to avoid inflated scores. section 12.3.5
















