How deeply does Deep Learning with R, Third Edition cover low-rank adaptation?

Explains frozen kernels, low-rank updates, parameter counts, and selective attention projections.

Low-rank adaptation

As explained in Deep Learning with R, Third Edition

Low-rank adaptation is a parameter-efficient fine-tuning technique that freezes a model's original weights and trains a low-rank update to selected projections.

Low-rank adaptation (LoRA) reduces fine-tuning memory use by freezing a model’s original weights and training a compact low-rank update to selected projections. section 16.3.3

A smaller update

For a 2,048 × 2,048 kernel, the frozen weights number 4,194,304. A rank-8 update uses two smaller matrices with 32,768 parameters in total. In KerasHub, LoRA can be enabled on a model backbone with `enable_lora(rank = 8L)`. section 16.3.3

Listing 16.13 Enabling LoRA training for a KerasHub model

gemma_lm$backbone$enable_lora(rank = 8L)

See the KerasHub call for enabling LoRA on a model backbone. section 16.3.3

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

LLMs in ProductionLow-rank adaptation is a parameter-efficient fine-tuning method that adapts a pretrained model through trainable low-rank updates while leaving most original parameters fixed.
liveBook manning.com

Building Reliable AI SystemsLow-Rank Adaptation is a parameter-efficient fine-tuning method that trains added low-rank matrices while keeping a pretrained model's weights frozen.
liveBook manning.com

Introduction to Generative AI, Second EditionLow-rank adaptation is a parameter-efficient fine-tuning technique that trains low-rank update matrices while keeping most pretrained parameters fixed.
liveBook manning.com

Generative AI in ActionLow-rank adaptation is a parameter-efficient technique that adapts a pretrained model by training small low-rank matrices while leaving the original model weights frozen.
liveBook manning.com

Build a Large Language Model (From Scratch)A parameter-efficient fine-tuning method that adapts a model by training low-rank updates instead of all weights.
liveBook manning.com

Transformers in ActionA low-rank adapter is a parameter-efficient fine-tuning component that represents a weight update through low-rank factors.
liveBook manning.com

The Complete Obsolete Guide to Generative AILoRA is a parameter-efficient fine-tuning technique that adapts a pretrained model by learning low-rank updates rather than modifying all model parameters.
liveBook manning.com

The AI Pocket BookA fine-tuning method that adds small trainable layers or low-rank updates while leaving the original model largely intact.

Compare how 9 books explain low-rank adaptation →

Related concepts

parameter-efficient fine-tuningkind of · Parameter-efficient fine-tuning adapts a pretrained model by training a limited subset of its parameters or added components rather than all its parameters.
QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
low-rank approximationuses · A low-rank approximation represents a matrix by a lower-rank matrix or factorization, often retaining important structure while reducing storage or computation.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.