liveBook · Manning

Low-rank adaptation

As explained in Domain-Specific Small Language Models

Low-rank adaptation parameters used to modify a pretrained model with a small number of additional learned weights.

Low-rank adaptation (LoRA) adapts a pretrained model by keeping its original weights frozen and fine-tuning only a subset of parameters. As a parameter-efficient fine-tuning (PEFT) method, it reduces computational cost compared with full fine-tuning while maintaining comparable accuracy and robustness. section 2.4 section 2.6

A practical adaptation workflow can include:

  • Load a pretrained model and prepare task-specific data. section 2.4
  • Configure LoRA, prepare the model for low-bit training, and attach the adapter. section 2.4
  • Train, save the adapter, then load it with the base model for inference. section 2.4

Dialogue summarization

A worked example adapts FLAN-T5-small, an 80-million-parameter encoder–decoder model, on the SAMSum dialogue-and-summary dataset. The setup uses 8-bit loading and targets the q and v modules with a LoRA configuration before training and generating a sample summary. section 2.4

Figure 2.3 The LoRA fine-tuning process

The process diagram highlights the central distinction: pretrained weights stay frozen while a subset of parameters is fine-tuned. section 2.4

How Domain-Specific Small Language Models covers it

How deep it goes
Walks through configuration, training, saving, loading, and inference, without detailing the low-rank mathematics.
Examples
Demonstrates summarization with FLAN-T5-small and SAMSum using an 8-bit model and LoRA adapters.
What it assumes
Assumes a pretrained model and task dataset, while demonstrating their loading, preprocessing, and adapter setup.
What you can do after
Apply it via PEFT to train and use a LoRA-adapted sequence-to-sequence model for dialogue summarization.
How it connects
Contrasts LoRA with full fine-tuning and places it among PEFT methods such as prefix tuning and AdaLoRA.

Also explained in

Compare how 10 books explain low-rank adaptation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.