A model parameter is a weight or bias learned from training data. Attention weights are one example. Parameters differ from hyperparameters: a learning rate is chosen before training, while parameter values change as the model learns. section 8.1 section 8.1.1
Gradient descent adjusts parameters to reduce a loss function. It calculates the loss gradient for each parameter, then moves that parameter in the opposite direction. The learning rate controls the size of the move; batch size affects the data used to calculate gradients, and the number of epochs controls how many passes training makes through the dataset. section 8.1.1
Not every parameter must change during adaptation. Low-rank adaptation (LoRA) keeps a pretrained weight matrix W₀ fixed and learns an update represented as BA, so the adapted weight is W₀ + BA. Here B has shape d × r and A has shape r × k for an original d × k matrix. Choosing a small rank r reduces the number of trainable parameters compared with updating the full matrix. section 8.3.1
Listing 8.11 Illustrative example of LoRA
def lora_decomposition(W, r): B = np.zeros((W.shape[0], r)) #1 A = np.random.normal(0, 1, (r, W.shape[1])) U, S, Vt = np.linalg.svd(W, full_matrices=False) #2 B[:, :r] = U[:, :r] * np.sqrt(S[:r]) A[:r, :] = np.sqrt(S[:r])[:, np.newaxis] * Vt[:r, :] W_approx = B @ A #3 return B, A, W_approx d = 6 #4 W = np.random.randint(0, 10, size=(d, d)) #5 r = 5 # Rank for the decomposition #6 B, A, W_approx = lora_decomposition(W, r) #7
The listing constructs B and A and multiplies them to approximate a matrix W. It uses singular value decomposition to illustrate a low-rank approximation; during LoRA training, B and A are learned instead. section 8.3.1
Listing 8.12 Applying LoRA with the Unsloth library
max_seq_length = 2048 lora_rank = 32 #1 model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Qwen3-4B-Base", max_seq_length = max_seq_length, load_in_4bit = False, #2 fast_inference = True, #3 max_lora_rank = lora_rank, gpu_memory_utilization = 0.7, ) model = FastLanguageModel.get_peft_model( model, r = lora_rank, target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_alpha = lora_rank*2, use_gradient_checkpointing = "unsloth", random_state = 3407, )
This Unsloth configuration sets a LoRA rank and names transformer projection modules to adapt, connecting the parameter-efficient update to a concrete model setup. section 8.3.1
