Adapter training specializes a pretrained model by adding small trainable modules while keeping the original weights frozen. Rather than producing a separate full-model checkpoint for each task, it preserves one shared foundation model and stores compact task or domain adaptations alongside it. This lowers training, storage, and deployment cost while retaining the base model’s general language capability. section 2.2 section 3.5
LoRA is a common adapter-training approach. It learns low-rank additions to selected weight matrices instead of directly updating the large pretrained matrices. The adapter’s rank controls learning capacity and parameter cost; alpha scales its effect; dropout helps limit overfitting; and target modules determine where adaptation occurs. Attention projections such as query, key, value, and output projections are practical targets, with feed-forward projections optionally included. section 2.1.2 section 3.5.1
Use adapter training when specialization must fit operational constraints:
- Maintain a single base model with separate finance, customer-support, or other domain adapters, swapping the active adapter without altering the shared backbone. section 2.2
- Choose rank by trading faster, smaller training against capacity for more complex adaptation; evaluate alternative LoRA settings rather than treating defaults as final. section 2.1.2
- Pair training inputs with clear expected outputs, then evaluate against predefined quality thresholds; dataset cleaning and evaluation form a repeated feedback loop. section 2.7
- Prefer a foundation model with relevant domain exposure when possible, reducing the data and compute needed for specialization. section 2.7
Finance and casual-response adapters
Create two LoRA configurations for the same T5 base model, save each adapter independently, and load one at a time for a prompt such as “summarize: Company ABC posted record profits in Q2 despite inflation pressures.” The selected adapter changes the model’s behavior without replacing the base model. section 2.2
Credit-risk adaptation
Attach LoRA modules to a quantized Llama model, format credit-risk records as instruction, input, and response examples, and train the adapters. A test applicant record containing income, debt, credit score, and late payments produces a High Risk classification. section 2.1.2
Figure 3.10 LoRA architecture, where the low-rank matrices A and B are fine-tuned while the pre-trained model remains untouched.

The diagram shows the core separation: low-rank matrices A and B are trained while the pretrained model remains unchanged. section 3.5.1



