Low-rank adaptation

Q: How deeply does Introduction to Generative AI, Second Edition cover low-rank adaptation?

Defines low-rank updates and fixed pretrained parameters but stops short of implementation details or training procedures.

As explained in Introduction to Generative AI, Second Edition

Low-rank adaptation is a parameter-efficient fine-tuning technique that trains low-rank update matrices while keeping most pretrained parameters fixed.

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique for customizing large language models. It trains low-rank update matrices while keeping most pretrained parameters fixed, reducing the barrier to model customization. section 11.4

LoRA belongs to a broader shift toward accessible open-model development, alongside QLoRA and lightweight alignment methods. Its efficiency can widen participation, while customization still requires infrastructure and responsible model governance. section 11.4

Table 11.2 Comparing open and closed LLMs

Trade-offs	Open source LLMs	Closed source LLMs
Transparency	Full access to weights, architecture, and sometimes data	Opaque methods, weights, and training data
Accessibility	Downloadable, runnable offline, and modifiable	API access controlled by the provider
Safety alignment	Varies widely; depends on developer practices and community norms	Extensive internal testing and standardized safeguards
License clarity	Apache, MIT, RAIL, custom terms; sometimes vague or restrictive	Proprietary terms of service
Customization	Fully customizable via fine-tuning and architectural changes	Limited to API parameters or provider tooling
Infrastructure burden	Requires local compute, setup, and deployment	Hosted by the provider
Risk of misuse	Higher; fewer enforced safeguards, and fewer default protections	Moderated through platform enforcement and terms of service

Compare customization access, infrastructure demands, transparency, and misuse risks for open and closed language models. section 11.4

How Introduction to Generative AI, Second Edition covers it

Cover of Introduction to Generative AI, Second Edition

liveBook manning.com

Read Introduction to Generative AI, Second Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

LLMs in ProductionLow-rank adaptation is a parameter-efficient fine-tuning method that adapts a pretrained model through trainable low-rank updates while leaving most original parameters fixed.
liveBook manning.com

Building Reliable AI SystemsLow-Rank Adaptation is a parameter-efficient fine-tuning method that trains added low-rank matrices while keeping a pretrained model's weights frozen.
liveBook manning.com

Generative AI in ActionLow-rank adaptation is a parameter-efficient technique that adapts a pretrained model by training small low-rank matrices while leaving the original model weights frozen.
liveBook manning.com

Build a Large Language Model (From Scratch)A parameter-efficient fine-tuning method that adapts a model by training low-rank updates instead of all weights.
liveBook manning.com

Transformers in ActionA low-rank adapter is a parameter-efficient fine-tuning component that represents a weight update through low-rank factors.
liveBook manning.com

The Complete Obsolete Guide to Generative AILoRA is a parameter-efficient fine-tuning technique that adapts a pretrained model by learning low-rank updates rather than modifying all model parameters.
liveBook manning.com

Deep Learning with R, Third EditionLow-rank adaptation is a parameter-efficient fine-tuning technique that freezes a model's original weights and trains a low-rank update to selected projections.
liveBook manning.com

The AI Pocket BookA fine-tuning method that adds small trainable layers or low-rank updates while leaving the original model largely intact.

Compare how 9 books explain low-rank adaptation →

Related concepts

parameter-efficient fine-tuningkind of · Parameter-efficient fine-tuning adapts a pretrained model by training a limited subset of its parameters or added components rather than all its parameters.
QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
low-rank approximationuses · A low-rank approximation represents a matrix by a lower-rank matrix or factorization, often retaining important structure while reducing storage or computation.

This explainer was generated by Manning's knowledge graph from the text of Introduction to Generative AI, Second Edition. The section links point to the parts of the book it draws on.