Parameter-efficient fine-tuning (PEFT) adapts a pretrained model without retraining its original weights. Instead, it keeps the base model frozen and adds a small set of trainable parameters, such as adapters or low-rank update matrices. This can specialize large models with a fraction of the compute and memory required by full fine-tuning while preserving broad pretrained capabilities. section 3.5 section 1.4.2
PEFT differs from partial fine-tuning. Partial approaches choose existing layers or parameters to update; PEFT introduces new trainable components alongside an entirely frozen backbone. The distinction matters when even selective updates to a billion-parameter model remain too costly. section 3.4 section 3.5
Common PEFT choices include:
- LoRA: inserts low-rank matrices into targeted layers. The matrices carry the task-specific update while the original weights remain unchanged. section 3.5.1
- QLoRA: combines LoRA with quantization to lower memory needs, making fine-tuning more accessible on consumer-grade hardware. section 1.4.2
- DoRA and NEFT: extend efficient adaptation through magnitude-direction decomposition and controlled embedding noise, respectively. section 3.8
LoRA is a practical default: it trains paired low-rank matrices A and B so their product approximates a weight update, then adds that update to the frozen layer output. Rank controls the adapter bottleneck; target modules determine which projections receive adapters. In one configuration, adapters are attached to attention projections such as q_proj, k_proj, and v_proj. section 3.5.1 section 2.1.2
Figure 2.2 LoRA adapter swapping for different domains or use cases. A single frozen foundation model serves as the shared backbone for multiple lightweight domain adapters. Each adapter adds a small set of trainable weights alongside the frozen layers — less than 0.1% of the base model's parameters — and can be swapped in or out without modifying the underlying model.

A shared frozen foundation model can serve multiple domains by loading a different lightweight adapter for each use case, rather than maintaining a separate full checkpoint. section 2.2
Domain adapters on one base model
A single Llama 72B base model can retain separate finance and customer-support adapters. Swap in the finance adapter for financial summaries or the customer-support adapter for empathetic responses, without changing the shared base model. Small adapters can be stored, compared, deployed, and potentially combined more easily than full specialized checkpoints. section 2.2
Credit-risk adaptation with LoRA
A 4-bit LLaMA-3 8B model is adapted with LoRA on a credit-risk dataset, then given an applicant’s income, debt, credit score, and late-payment history. The resulting response classifies the applicant as high risk, illustrating domain-specific learning on limited GPU resources. section 2.1.2







