Selective layer freezing fine-tunes a pretrained neural network by keeping chosen layers fixed while allowing the remaining layers to update. It reduces the optimizer’s effective parameter space, concentrates gradient flow on adaptation-sensitive regions, lowers computation, and helps preserve established representations. The choice is flexible: freeze lower layers, upper layers, roughly half the network, individual sublayers, or selected parameters. section 3.4.2 section 3.4
Choose a freezing strategy according to the task, data, and compute budget:
- Use static freezing when the trainable subset can be chosen before training; use dynamic schedules when layers should be unfrozen according to validation loss or gradient behavior. section 3.4.2
- Use magnitude-based, variance-based, or sensitivity-based freezing to identify layers that contribute little learning signal. section 3.4.3
- Match the task structure: preserve embeddings and lower transformer layers for semantic adaptation, or partially unfreeze lower layers for fine-grained syntactic adjustments. section 3.4.3
- Apply freezing at sublayer or parameter granularity, such as keeping an attention output projection fixed while training other components. section 3.4.2 section 3.4.4
BERT text classification
Load a pretrained BERT-base sequence-classification model with two labels, tokenize labeled text samples, and freeze the bottom six layers of its 12-layer encoder. Keep the upper layers trainable, optionally freeze their attention output projections, and pass only parameters with requires_grad enabled to AdamW. After the third epoch, unfreeze the upper layers, rebuild the optimizer, and inspect each layer’s final trainable status. section 3.4.4
Shows a time-dependent mask that switches a layer from frozen to trainable after its unfreezing threshold. section 3.4.2




