Low-rank adaptation
Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
LLMs in Production
Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a pretrained model through trainable low-rank updates while leaving most original parameters fixed.
- How it is used
- Adapts a pretrained model to a specific task or domain without retraining it from scratch.
- How deep it goes
- Works through low-rank update dimensions, rank trade-offs, PyTorch layer integration, training, and inference; defers a full backpropagation explanation.
- Examples
- Uses a 100 × 100 update factored at rank 2 and a PyTorch LoRA layer added to a linear layer.
- What you can do after
- Build it from scratch as a PyTorch layer, then load or merge its saved update for inference.
Building Reliable AI Systems
Low-Rank Adaptation is a parameter-efficient fine-tuning method that trains added low-rank matrices while keeping a pretrained model's weights frozen.
- How it is used
- Uses LoRA to adapt large pretrained models when full fine-tuning is too memory- or compute-intensive.
- How deep it goes
- Explains frozen weights and trainable matrices, then configures adapters in PEFT without deriving the underlying matrix mathematics.
- Examples
- Builds a LLaMA-2-7B customer-support assistant from Bitext instruction–response pairs and tests order, refund, and shipping prompts.
- What you can do after
- Apply it via PEFT and SFTTrainer to train, evaluate, save, and reload adapters for a pretrained conversational model.
Read how Building Reliable AI Systems explains low-rank adaptation →
Introduction to Generative AI, Second Edition
Low-rank adaptation is a parameter-efficient fine-tuning technique that trains low-rank update matrices while keeping most pretrained parameters fixed.
- How it is used
- Lowers the barrier to customizing large language models within the open-source model ecosystem.
- How deep it goes
- Defines low-rank updates and fixed pretrained parameters but stops short of implementation details or training procedures.
- Examples
- Uses customization of large open language models as its concrete application.
- What you can do after
- Apply it to customize large language models without updating most pretrained parameters.
Read how Introduction to Generative AI, Second Edition explains low-rank adaptation →
Generative AI in Action
Low-rank adaptation is a parameter-efficient technique that adapts a pretrained model by training small low-rank matrices while leaving the original model weights frozen.
- How it is used
- Adapts pretrained language models to specific tasks while updating fewer parameters than full fine-tuning.
- How deep it goes
- Explains frozen pretrained weights and trainable low-rank matrices through a layer diagram, without a rank-selection procedure.
- Examples
- Uses a layer diagram labeling input X, frozen weight W, and trainable matrices A and B.
- What you can do after
- Evaluate/understand it as a lower-resource adaptation option while planning task-specific evaluation before deployment.
Read how Generative AI in Action explains low-rank adaptation →
Build a Large Language Model (From Scratch)
A parameter-efficient fine-tuning method that adapts a model by training low-rank updates instead of all weights.
- How it is used
- Applies parameter-efficient updates when instruction-fine-tuning a language model more efficiently.
- How deep it goes
- Defines low-rank updates and assigns a runtime-versus-performance experiment, but does not detail the underlying implementation.
- Examples
- Uses a code-modification exercise comparing training runtime and model performance before and after LoRA.
- What you can do after
- Apply it via an existing fine-tuning implementation and evaluate its runtime and model-performance effects.
Read how Build a Large Language Model (From Scratch) explains low-rank adaptation →
Transformers in Action
A low-rank adapter is a parameter-efficient fine-tuning component that represents a weight update through low-rank factors.
- How it is used
- Adapts pretrained transformer models to downstream tasks while training a small subset of parameters.
- How deep it goes
- Explains frozen weights, factor dimensions, rank constraints, SVD intuition, and library configuration, but not the training loop.
- Examples
- Uses NumPy SVD decomposition, matrix visualizations, and an Unsloth configuration targeting Qwen3-4B-Base.
- What you can do after
- Apply it via an Unsloth PEFT interface by selecting a rank and transformer target modules.
Read how Transformers in Action explains low-rank adaptation →
The Complete Obsolete Guide to Generative AI
LoRA is a parameter-efficient fine-tuning technique that adapts a pretrained model by learning low-rank updates rather than modifying all model parameters.
- How it is used
- Positions LoRA as a resource-efficient option when deciding whether to fine-tune an existing LLM for a particular use case.
- How deep it goes
- Introduces LoRA through its efficiency and hardware implications, while deferring practical implementation and low-rank mechanics.
- Examples
- Uses Nvidia 3090 hardware and possible multi-GPU setups as the concrete LoRA fine-tuning examples.
- What you can do after
- Evaluate/understand LoRA as a lower-resource fine-tuning option for customizing an existing model with your own data.
Read how The Complete Obsolete Guide to Generative AI explains low-rank adaptation →
Deep Learning with R, Third Edition
Low-rank adaptation is a parameter-efficient fine-tuning technique that freezes a model's original weights and trains a low-rank update to selected projections.
- How it is used
- Reduces accelerator-memory use when fine-tuning large language models.
- How deep it goes
- Explains frozen kernels, low-rank updates, parameter counts, and selective attention projections.
- Examples
- Uses a 2,048-square kernel, a rank-8 update, and KerasHub model configuration.
- What you can do after
- Apply LoRA through KerasHub or enable it on selected attention projections.
Read how Deep Learning with R, Third Edition explains low-rank adaptation →
The AI Pocket Book
A fine-tuning method that adds small trainable layers or low-rank updates while leaving the original model largely intact.
- How it is used
- Limits model updates during fine-tuning to improve training efficiency and reduce overfitting risk.
- How deep it goes
- Explains inserted learnable layers, preserved original parameters, update savings, and overfitting control without mathematical detail.
- Examples
- Uses inserted small layers, new learnable parameters, and unchanged original model parameters as its central artifacts.
- What you can do after
- Apply LoRA to fine-tune an existing model while monitoring validation or test performance for overfitting.
Related concepts
- parameter-efficient fine-tuningkind of · Parameter-efficient fine-tuning adapts a pretrained model by training a limited subset of its parameters or added components rather than all its parameters.
- QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
- low-rank approximationuses · A low-rank approximation represents a matrix by a lower-rank matrix or factorization, often retaining important structure while reducing storage or computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.