To freeze a layer is to keep its weights unchanged during further training. This lets a pretrained convolutional base supply image features while newly added classifier layers learn a new task. Without freezing, large updates from a randomly initialized classifier could disrupt the base’s learned representations. section 8.3.1
Freeze a pretrained image backbone
In R with KerasHub, create the backbone with `trainable = FALSE`: `conv_base <- keras_hub$models$Backbone$from_preset("xception_41_imagenet", trainable = FALSE)`. Connect it to a new classifier, then compile and fit the combined model. Only the added Dense layers’ weights train. This arrangement can use image data augmentation during training, though running the backbone for each batch is more expensive than extracting features ahead of time. section 8.3.1
Two practical checks:
- Set a layer or model’s `trainable` attribute to `FALSE`, or call `freeze_weights()`. Freezing removes its weights from the trainable-weight list; the illustrated backbone goes from 154 trainable weights to zero. section 8.3.1
- Compile after changing trainability. If you freeze or unfreeze weights after compilation, recompile or the change will be ignored during training. section 8.3.1
Freezing also saves accelerator memory: parameters that never update need no optimizer variables. Low-Rank Adaptation (LoRA) uses this idea for large language model fine-tuning. It freezes a projection’s kernel and trains two much smaller matrices that supply an update. In the illustrated case, a 2,048 × 2,048 kernel has 4,194,304 frozen parameters, while a rank-eight update has 32,768 trainable parameters. section 16.3.3
layer_lora_linear <- new_layer_class(
classname = "LoraLinear",
initialize = function(input_dim, output_dim, rank) {
super$initialize()
self$kernel <- self$add_weight(shape(input_dim, output_dim),
trainable = FALSE)
self$alpha <- self$add_weight(shape(input_dim, rank))
self$beta <- self$add_weight(shape(rank, output_dim))
},
call = function(inputs) {
frozen <- inputs %*% self$kernel
update <- inputs %*% self$alpha %*% self$beta
frozen + update
}
)
The `LoraLinear` code marks `kernel` as non-trainable, then adds the trainable `alpha` and `beta` projections to its output. section 16.3.3
