Low-rank adaptation

As explained in Generative AI in Action

Low-rank adaptation is a parameter-efficient technique that adapts a pretrained model by training small low-rank matrices while leaving the original model weights frozen.

Low-rank adaptation (LoRA) specializes a pretrained large language model without updating all its weights. It is a parameter-efficient fine-tuning method: instead of retraining the full model, it trains a much smaller set of added parameters. This makes adapting large models more practical when computing resources are limited. section 9.1.1 section 9.6

LoRA adds small, low-rank matrices to model layers, including attention and feed-forward layers. During training, the original pretrained weights remain frozen while the added matrices are updated. In the layer diagram, W is the frozen weight, X is the input, and A and B are the trainable adaptation parameters. section 9.6.1

Figure 9.14 LoRA reparameterization—only A and B are trained [9]

Use the diagram to distinguish the frozen pretrained weight W from the trainable adaptation parameters A and B. section 9.6.1

For a task-specific adaptation, the main practical trade-offs are:

• Train and store fewer parameters than with full-model fine-tuning; task-specific parameters can be loaded alongside the pretrained model. section 9.6.1
• Keeping pretrained weights fixed helps retain general capabilities, but it does not remove the need to evaluate the adapted model. section 9.6 section 9.8
• Plan for task-specific data, repeated evaluation, and ethical considerations before production deployment. section 9.8

How Generative AI in Action covers it

How deep it goes

Explains frozen pretrained weights and trainable low-rank matrices through a layer diagram, without a rank-selection procedure.

Examples

Uses a layer diagram labeling input X, frozen weight W, and trainable matrices A and B.

What it assumes

Introduces the roles of pretrained weights and added matrices, using basic model-layer terminology without a mathematical derivation.

What you can do after

Evaluate/understand it as a lower-resource adaptation option while planning task-specific evaluation before deployment.

How it connects

Relates LoRA to full fine-tuning, parameter-efficient fine-tuning, and catastrophic forgetting.

liveBookmanning.com

Read Generative AI in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionLow-rank adaptation is a parameter-efficient fine-tuning method that adapts a pretrained model through trainable low-rank updates while leaving most original parameters fixed.
• 

  liveBookmanning.com
  Building Reliable AI SystemsLow-Rank Adaptation is a parameter-efficient fine-tuning method that trains added low-rank matrices while keeping a pretrained model's weights frozen.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionLow-rank adaptation is a parameter-efficient fine-tuning technique that trains low-rank update matrices while keeping most pretrained parameters fixed.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A parameter-efficient fine-tuning method that adapts a model by training low-rank updates instead of all weights.
• 

  liveBookmanning.com
  Transformers in ActionA low-rank adapter is a parameter-efficient fine-tuning component that represents a weight update through low-rank factors.
• 

  liveBookmanning.com
  The Complete Obsolete Guide to Generative AILoRA is a parameter-efficient fine-tuning technique that adapts a pretrained model by learning low-rank updates rather than modifying all model parameters.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionLow-rank adaptation is a parameter-efficient fine-tuning technique that freezes a model's original weights and trains a low-rank update to selected projections.
• 

  liveBookmanning.com
  The AI Pocket BookA fine-tuning method that adds small trainable layers or low-rank updates while leaving the original model largely intact.

Compare how 9 books explain low-rank adaptation →

Related concepts

• parameter-efficient fine-tuningkind of · Parameter-efficient fine-tuning adapts a pretrained model by training a limited subset of its parameters or added components rather than all its parameters.
• QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
• low-rank approximationuses · A low-rank approximation represents a matrix by a lower-rank matrix or factorization, often retaining important structure while reducing storage or computation.

This explainer was generated by Manning's knowledge graph from the text of Generative AI in Action. The section links point to the parts of the book it draws on.