A hyperparameter is a setting chosen before training that controls how a model learns or how it is built. It differs from a model parameter: gradient descent learns weights and biases from data, but the learning rate governing those updates is set in advance. Both affect how the model minimizes its loss. section 8.1 section 8.1.1
Common choices have different practical consequences:
- Learning rate sets the update step size. Too high can overshoot a minimum; too low can make training slow or leave it stuck. section 8.1.1
- Batch size determines how many data points contribute to each gradient calculation, affecting convergence stability and speed. section 8.1.1
- Epoch count determines how often training passes through the dataset. More passes can improve learning, but eventually risk overfitting; early stopping uses a validation metric to interrupt training when improvement stops. section 8.1.1
- Architecture settings such as layer count, attention-head count, and hidden and embedding dimensions shape model capacity, expressiveness, and inference latency. section 3.2
Rank as a fine-tuning hyperparameter
In low-rank adaptation (LoRA), a chosen rank r controls the size of the learned low-rank weight update. The pretrained weight matrix stays fixed while the update is represented by two smaller matrices. Here r is configured; the matrices are learned. section 8.3.1
Listing 8.12 Applying LoRA with the Unsloth library
max_seq_length = 2048 lora_rank = 32 #1 model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Qwen3-4B-Base", max_seq_length = max_seq_length, load_in_4bit = False, #2 fast_inference = True, #3 max_lora_rank = lora_rank, gpu_memory_utilization = 0.7, ) model = FastLanguageModel.get_peft_model( model, r = lora_rank, target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_alpha = lora_rank*2, use_gradient_checkpointing = "unsloth", random_state = 3407, )
The Unsloth listing sets lora_rank to 32, then passes it into both model loading and LoRA configuration. It shows a hyperparameter as an explicit configuration value rather than a learned weight. section 8.3.1
Choosing values is an experimentation task, not a training update. Ray Data and Tune can organize and parallelize a hyperparameter search; Weights & Biases can track experiments and run sweeps so results can be compared. section 8.4
