liveBook · Manning

Low-rank adaptation

As explained in Build AI Drug Discovery Pipelines

LoRA is a parameter-efficient fine-tuning method that learns low-rank updates while keeping most pretrained weights fixed.

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning method: it learns low-rank updates while keeping most pretrained weights fixed. By updating only a small fraction of parameters, it can achieve competitive performance on classification tasks. section 12.3.5

For protein classification, a pretrained ESM-2 model can produce an embedding for each sequence position. Mean pooling combines these embeddings into a sequence representation, which a classification head uses to predict a label. LoRA offers a way to adapt the pretrained model without training its entire backbone. section 12.3.5

Listing 12.6 Fine-tuning ESM-2 with a classification head (optional, hands-on)
device = "cuda" if torch.cuda.is_available() else "cpu"
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
backbone = AutoModel.from_pretrained(model_name)
 
class ESM2Classifier(nn.Module):
    def __init__(self, backbone, num_classes, freeze_backbone=True):
        super().__init__()
        self.backbone = backbone
        hidden = backbone.config.hidden_size
        if freeze_backbone:
            for p in self.backbone.parameters():
                p.requires_grad = False
        self.head = nn.Linear(hidden, num_classes)
 
    def mean_pool(self, hidden_states, attention_mask):
        mask = attention_mask.unsqueeze(-1).float()  # (B, L, 1) 
        summed = (hidden_states * mask).sum(dim=1)
        counts = mask.sum(dim=1).clamp(min=1e-9) 
        return summed / counts
 
    def forward(self, input_ids, attention_mask):
        out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
        pooled = self.mean_pool(out.last_hidden_state, attention_mask)
        return self.head(pooled)  # (B, num_classes)
 
model = ESM2Classifier(backbone, num_classes=2, freeze_backbone=True).to(device)
optimizer = torch.optim.Adam(
    (p for p in model.parameters() if p.requires_grad), lr=1e-4)
loss_fn = nn.CrossEntropyLoss()
 
def run_epoch(loader, train=True):  # Loader yields (sequences, labels)
    model.train() if train else model.eval()
    total = 0.0
    with torch.set_grad_enabled(train):
        for sequences, labels in loader:
            enc = tokenizer(list(sequences), padding=True, return_tensors="pt")
            input_ids = enc["input_ids"].to(device)
            attention_mask = enc["attention_mask"].to(device)
            labels = torch.tensor(labels, dtype=torch.long, device=device)
            logits = model(input_ids, attention_mask)
            loss = loss_fn(logits, labels)
            if train:
                optimizer.zero_grad()
                loss.backward()
                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
                optimizer.step()
            total += loss.item() * input_ids.size(0)
    return total / len(loader.dataset)

This classifier illustrates two alternatives to LoRA: training only the classification head or fine-tuning the whole backbone. It does not implement LoRA. section 12.3.5

For antibody classification, keep homologous sequences out of both the training and test sets. Cluster by sequence identity, or by clonotype and donor, and assign whole clusters to one side of the split to avoid inflated scores. section 12.3.5

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines the parameter-efficiency motivation but does not demonstrate a LoRA implementation.
Examples
Contrasts head-only training and full-backbone fine-tuning in an ESM-2 classifier; neither example implements LoRA.
What you can do after
Evaluate LoRA as a parameter-efficient alternative to full-backbone fine-tuning for classification.
How it connects
Connects LoRA with fine-tuning, protein embeddings, mean pooling, and classification heads.

Also explained in

Compare how 17 books explain low-rank adaptation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.