Low-rank adaptation
Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
LLM Customization and Fine-Tuning
A learned, low-rank additive change to a model's weights.
- How it is used
- Uses LoRA as a lower-cost adaptation step when prompting falls short, before considering full supervised fine-tuning.
- How deep it goes
- Explains frozen weights and low-rank matrix updates, then walks through adapter training, evaluation, safety checks, and deployment tradeoffs.
- Examples
- Compares an IT support adapter with prompted baselines on six held-out questions and trains a Qwen3-4B adapter in a single-GPU quickstart.
- What you can do after
- Apply it via Hugging Face PEFT and TRL to train, save, and evaluate an adapter, including checks for safety regressions.
Read how LLM Customization and Fine-Tuning explains low-rank adaptation →
LLMs in Production
Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a pretrained model through trainable low-rank updates while leaving most original parameters fixed.
- How it is used
- Adapts pretrained models to specific tasks or domains without retraining their original weights.
- How deep it goes
- Explains low-rank update matrices and rank trade-offs, then demonstrates a PyTorch adapter and inference options without a full backpropagation tutorial.
- Examples
- Uses a 100 × 100 matrix update, a PEFT token-classification setup, and a rank-16 Llama-style adapter.
- What you can do after
- Apply it via Hugging Face PEFT or build a PyTorch adapter, then load or merge it for inference.
Building Specialized AI Systems
A low-rank update is a parameter change represented as a product of lower-dimensional matrices to reduce adaptation cost.
- How it is used
- Specializes pretrained language models for domain tasks while limiting training compute, memory, and storage costs.
- How deep it goes
- Explains frozen weights, low-rank matrices, targeted layers, configuration, training examples, and method trade-offs without deriving optimization details.
- Examples
- Demonstrates credit-risk assessment, analyst-report summarization, customer-support classification, and Rotten Tomatoes sentiment classification.
- What you can do after
- Apply it via Unsloth or PEFT by attaching and configuring trainable adapters on selected pretrained-model layers.
Read how Building Specialized AI Systems explains low-rank adaptation →
Local Image Generation
A low-rank adaptation component that modifies a pretrained model's behavior using a relatively small set of trainable parameters.
- How it is used
- Adds styles or custom subjects to Flux and Stable Diffusion outputs without full-model fine-tuning.
- How deep it goes
- Explains attention-layer modification, then provides workflow wiring, strength tuning, data collection, training setup, and checkpoint evaluation.
- Examples
- Uses a Kafka gate scene, an Amateur Photography style LoRA, and a Keats subject LoRA trained from varied dog photos.
- What you can do after
- Apply it via ComfyUI and AI-Toolkit to alter image styles, train subject LoRAs, and select useful checkpoints.
Read how Local Image Generation explains low-rank adaptation →
Build AI Drug Discovery Pipelines
LoRA is a parameter-efficient fine-tuning method that learns low-rank updates while keeping most pretrained weights fixed.
- How it is used
- Introduces LoRA as a parameter-efficient option for adapting a pretrained protein model to classification.
- How deep it goes
- Defines the parameter-efficiency motivation but does not demonstrate a LoRA implementation.
- Examples
- Contrasts head-only training and full-backbone fine-tuning in an ESM-2 classifier; neither example implements LoRA.
- What you can do after
- Evaluate LoRA as a parameter-efficient alternative to full-backbone fine-tuning for classification.
Read how Build AI Drug Discovery Pipelines explains low-rank adaptation →
Rearchitecting LLMs
Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains low-rank updates while keeping the original model weights frozen.
- How it is used
- Specializes recovered models for domain tasks or structured function-calling behavior using parameter-efficient fine-tuning.
- How deep it goes
- Explains matrix shapes and parameter savings, then walks through adapter configuration, training, and merging.
- Examples
- Calculates SmolLM2 attention-projection savings and configures Qwen3-0.6B for function calling.
- What you can do after
- Apply it via PEFT to configure, train, save, and merge a function-calling adapter.
Building Reliable AI Systems
Low-Rank Adaptation is a parameter-efficient fine-tuning method that trains added low-rank matrices while keeping a pretrained model's weights frozen.
- How it is used
- Reduces the resource cost of adapting a pretrained model for tasks such as customer support.
- How deep it goes
- Explains frozen weights and trainable matrices, then configures and trains adapters in a QLoRA workflow.
- Examples
- Uses LLaMA-2-7B and a 1,000-example Bitext customer-support dataset to demonstrate QLoRA.
- What you can do after
- Apply LoRA through PEFT configuration and train, test, save, and reload a QLoRA adapter.
Read how Building Reliable AI Systems explains low-rank adaptation →
Introduction to Generative AI, Second Edition
Low-rank adaptation is a parameter-efficient fine-tuning technique that trains low-rank update matrices while keeping most pretrained parameters fixed.
- How it is used
- Lowers the barrier to customizing large language models within the open-source model ecosystem.
- How deep it goes
- Defines low-rank updates and fixed pretrained parameters but stops short of implementation details or training procedures.
- Examples
- Uses customization of large open language models as its concrete application.
- What you can do after
- Apply it to customize large language models without updating most pretrained parameters.
Read how Introduction to Generative AI, Second Edition explains low-rank adaptation →
Generative AI in Action
Low-rank adaptation is a parameter-efficient technique that adapts a pretrained model by training small low-rank matrices while leaving the original model weights frozen.
- How it is used
- Uses LoRA to adapt pretrained language models to tasks or specialized data with fewer trainable parameters than full fine-tuning.
- How deep it goes
- Explains frozen pretrained weights and trainable low-rank matrices in model layers, including their storage for downstream tasks.
- Examples
- Uses a layer diagram with frozen weight W and trainable matrices A and B to illustrate LoRA reparameterization.
- What you can do after
- Evaluate/understand LoRA’s resource trade-offs and the need to evaluate adapted models before production.
Read how Generative AI in Action explains low-rank adaptation →
Applied Reinforcement Learning
LoRA is a parameter-efficient fine-tuning technique that trains low-rank updates to a model's weights while keeping its original weights fixed.
- How it is used
- Reduces the memory burden of fine-tuning large language models by training low-rank updates while freezing original weights.
- How deep it goes
- Quantifies full-fine-tuning memory for weights, optimizer state, and activations, then frames LoRA as a low-memory alternative.
- Examples
- Uses financial reasoning and estimates the memory needed to fine-tune a 3-billion-parameter model on a 24-GB GPU.
- What you can do after
- Apply it to fine-tune large models with a tiny trainable parameter subset and target training on a single 24-GB GPU.
Read how Applied Reinforcement Learning explains low-rank adaptation →
Quantization and Fast Inference
LoRA is a parameter-efficient fine-tuning method that freezes pretrained weights and learns low-rank update matrices.
- How it is used
- Adapts pretrained models to domain-specific tasks without updating the full set of base weights.
- How deep it goes
- Explains the frozen-base and trainable-adapter roles, then gives a QLoRA configuration and measured memory costs.
- Examples
- Uses rank-16 adapters on Llama-2-7B and reports a 12.31 GB peak training footprint.
- What you can do after
- Apply it via bitsandbytes and PEFT to load an NF4 base and attach trainable adapters.
Read how Quantization and Fast Inference explains low-rank adaptation →
Build a Large Language Model (From Scratch)
A parameter-efficient fine-tuning method that adapts a model by training low-rank updates instead of all weights.
- How it is used
- Applies parameter-efficient updates when instruction-fine-tuning a language model more efficiently.
- How deep it goes
- Defines low-rank updates and assigns a runtime-versus-performance experiment, but does not detail the underlying implementation.
- Examples
- Uses a code-modification exercise comparing training runtime and model performance before and after LoRA.
- What you can do after
- Apply it via an existing fine-tuning implementation and evaluate its runtime and model-performance effects.
Read how Build a Large Language Model (From Scratch) explains low-rank adaptation →
Transformers in Action
A low-rank adapter is a parameter-efficient fine-tuning component that represents a weight update through low-rank factors.
- How it is used
- Adapts pretrained transformer models to downstream tasks while training a small subset of parameters.
- How deep it goes
- Explains frozen weights, factor dimensions, rank constraints, SVD intuition, and library configuration, but not the training loop.
- Examples
- Uses NumPy SVD decomposition, matrix visualizations, and an Unsloth configuration targeting Qwen3-4B-Base.
- What you can do after
- Apply it via an Unsloth PEFT interface by selecting a rank and transformer target modules.
Read how Transformers in Action explains low-rank adaptation →
The Complete Obsolete Guide to Generative AI
LoRA is a parameter-efficient fine-tuning technique that adapts a pretrained model by learning low-rank updates rather than modifying all model parameters.
- How it is used
- Positions LoRA as a resource-efficient option when deciding whether to fine-tune an existing LLM for a particular use case.
- How deep it goes
- Introduces LoRA through its efficiency and hardware implications, while deferring practical implementation and low-rank mechanics.
- Examples
- Uses Nvidia 3090 hardware and possible multi-GPU setups as the concrete LoRA fine-tuning examples.
- What you can do after
- Evaluate/understand LoRA as a lower-resource fine-tuning option for customizing an existing model with your own data.
Read how The Complete Obsolete Guide to Generative AI explains low-rank adaptation →
Deep Learning with R, Third Edition
Low-rank adaptation is a parameter-efficient fine-tuning technique that freezes a model's original weights and trains a low-rank update to selected projections.
- How it is used
- Reduces accelerator-memory use when fine-tuning large language models.
- How deep it goes
- Explains frozen kernels, low-rank updates, parameter counts, and selective attention projections.
- Examples
- Uses a 2,048-square kernel, a rank-8 update, and KerasHub model configuration.
- What you can do after
- Apply LoRA through KerasHub or enable it on selected attention projections.
Read how Deep Learning with R, Third Edition explains low-rank adaptation →
The AI Pocket Book
A fine-tuning method that adds small trainable layers or low-rank updates while leaving the original model largely intact.
- How it is used
- Limits model updates during fine-tuning to improve training efficiency and reduce overfitting risk.
- How deep it goes
- Explains inserted learnable layers, preserved original parameters, update savings, and overfitting control without mathematical detail.
- Examples
- Uses inserted small layers, new learnable parameters, and unchanged original model parameters as its central artifacts.
- What you can do after
- Apply LoRA to fine-tune an existing model while monitoring validation or test performance for overfitting.
Domain-Specific Small Language Models
Low-rank adaptation parameters used to modify a pretrained model with a small number of additional learned weights.
- How it is used
- Adapts pretrained models to specific tasks or domains without updating all model parameters.
- How deep it goes
- Walks through configuration, training, saving, loading, and inference, without detailing the low-rank mathematics.
- Examples
- Demonstrates summarization with FLAN-T5-small and SAMSum using an 8-bit model and LoRA adapters.
- What you can do after
- Apply it via PEFT to train and use a LoRA-adapted sequence-to-sequence model for dialogue summarization.
Read how Domain-Specific Small Language Models explains low-rank adaptation →
Also covered in
- Deep Learning with JAX
- Software Testing with Generative AI
- Deep Learning with Python, Third Edition
- Machines that ThinkIn early access (MEAP)
- Grokking AI Algorithms, Second Edition
What to know first
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- foundation modelA foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- parameter-efficient fine-tuningkind of · Parameter-efficient fine-tuning adapts a pretrained model by training a limited set of existing or added parameters while leaving most of the model unchanged.
- low-rank approximationuses · A low-rank approximation represents a matrix or transformation with a lower-rank matrix or factorization, often retaining dominant structure while reducing storage or computation.
- QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that combines a quantized pretrained model with trainable low-rank adapters.
- adapteruses · An adapter is a small trainable module added to a pretrained model to modify its behavior while leaving most of the model’s original parameters unchanged.
- PEFT libraryimplemented by · PEFT is a software library that provides methods and APIs for parameter-efficient adaptation and fine-tuning of pretrained machine-learning models.
Questions about low-rank adaptation
What is low-rank adaptation?
Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
Which Manning books explain low-rank adaptation?
LLM Customization and Fine-Tuning, LLMs in Production, Building Specialized AI Systems, Local Image Generation, Build AI Drug Discovery Pipelines, Rearchitecting LLMs, Building Reliable AI Systems, Introduction to Generative AI, Second Edition and 9 more.
What should I know before learning low-rank adaptation?
Manning books that teach low-rank adaptation usually introduce fine-tuning, language model, foundation model and parameter first.
What is low-rank adaptation a kind of?
Low-rank adaptation is a kind of parameter-efficient fine-tuning. Parameter-efficient fine-tuning adapts a pretrained model by training a limited set of existing or added parameters while leaving most of the model unchanged.
What is low-rank adaptation also called?
Low-rank adaptation is also called LoRA.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
















