Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that specializes a pretrained model without retraining its full parameter set. The original weights remain frozen; small trainable adapter matrices are inserted into selected layers, so the model can learn task-specific behavior while retaining its pretrained capabilities. LoRA commonly changes only a small fraction of parameters, reducing compute, memory, and storage demands compared with full fine-tuning. section 1.4.2 section 3.8
At a target linear layer, LoRA represents the weight update as two smaller matrices. Matrix A projects an input down to a low-rank bottleneck r, and matrix B projects it back to the original dimension; their product approximates the update ΔW. The layer combines the frozen pretrained path with this learned adapter path. For a 64×64 weight matrix, rank 4 adapters use 512 parameters rather than 4,096 for a full update. section 3.5.1
Figure 3.10 LoRA architecture, where the low-rank matrices A and B are fine-tuned while the pre-trained model remains untouched.

The diagram separates the unchanged pretrained pathway from the trainable A and B adapter pathway, showing how both contribute to the final output. section 3.5.1
Use LoRA to balance specialization against operational constraints:
- Target projection layers relevant to the task, then configure rank, scaling, dropout, bias handling, and the task type. section 3.5.1
- Start from a foundation model with useful domain exposure; this reduces the data and compute needed for specialization. section 2.7
- Pair consistent inputs with verified target outputs, train the adapters, and evaluate against predefined quality thresholds. section 2.7
- Consider QLoRA when additional memory reduction is needed through 4-bit quantization; weigh all methods against adaptation depth and preservation of pretrained knowledge. section 1.4.2 section 3.8
Credit-risk specialization on a single GPU
A 4-bit LLaMA-3 8B model is adapted with LoRA adapter layers using Unsloth, then trained on credit-risk input-output examples. A test case supplies income, debt, credit score, and late-payment data; the adapted model returns a High Risk assessment. The setup uses rank 16 and targets attention and feed-forward projection modules, illustrating a consumer-hardware path to domain-specific reasoning. section 2.1.2
Customer-support ticket prioritization
A fine-tuned GPT-3.5 model uses LoRA and 18 months of support history to classify priority and routing needs. It distinguishes routine billing questions from rare security breaches and outages, with reported gains in critical-incident F1, high-priority recall, response time, and support efficiency while using far fewer trainable parameters than traditional fine-tuning. section 1.4.3
LoRA is not a substitute for data quality or evaluation. Training should monitor quantitative measures such as loss and accuracy alongside qualitative output review, because specialization can otherwise fail to meet professional or organizational requirements. Full fine-tuning may offer greater specialization, but costs more and can risk catastrophic forgetting; LoRA instead favors modular, swappable adapters for different domains or tasks. section 1.4.2 section 2.7 section 3.8
















