Low-rank adaptation (LoRA) adapts a pretrained model by keeping its original weights frozen and fine-tuning only a subset of parameters. As a parameter-efficient fine-tuning (PEFT) method, it reduces computational cost compared with full fine-tuning while maintaining comparable accuracy and robustness. section 2.4 section 2.6
A practical adaptation workflow can include:
- Load a pretrained model and prepare task-specific data. section 2.4
- Configure LoRA, prepare the model for low-bit training, and attach the adapter. section 2.4
- Train, save the adapter, then load it with the base model for inference. section 2.4
Dialogue summarization
A worked example adapts FLAN-T5-small, an 80-million-parameter encoder–decoder model, on the SAMSum dialogue-and-summary dataset. The setup uses 8-bit loading and targets the q and v modules with a LoRA configuration before training and generating a sample summary. section 2.4
Figure 2.3 The LoRA fine-tuning process
The process diagram highlights the central distinction: pretrained weights stay frozen while a subset of parameters is fine-tuned. section 2.4









