liveBook · Manning

Freeze

As explained in Building Specialized AI Systems

Selective layer freezing is a fine-tuning technique that keeps chosen neural-network layers fixed while allowing other layers to update.

Selective layer freezing fine-tunes a pretrained neural network by keeping chosen layers fixed while allowing the remaining layers to update. It reduces the optimizer’s effective parameter space, concentrates gradient flow on adaptation-sensitive regions, lowers computation, and helps preserve established representations. The choice is flexible: freeze lower layers, upper layers, roughly half the network, individual sublayers, or selected parameters. section 3.4.2 section 3.4

Choose a freezing strategy according to the task, data, and compute budget:

  • Use static freezing when the trainable subset can be chosen before training; use dynamic schedules when layers should be unfrozen according to validation loss or gradient behavior. section 3.4.2
  • Use magnitude-based, variance-based, or sensitivity-based freezing to identify layers that contribute little learning signal. section 3.4.3
  • Match the task structure: preserve embeddings and lower transformer layers for semantic adaptation, or partially unfreeze lower layers for fine-grained syntactic adjustments. section 3.4.3
  • Apply freezing at sublayer or parameter granularity, such as keeping an attention output projection fixed while training other components. section 3.4.2 section 3.4.4

BERT text classification

Load a pretrained BERT-base sequence-classification model with two labels, tokenize labeled text samples, and freeze the bottom six layers of its 12-layer encoder. Keep the upper layers trainable, optionally freeze their attention output projections, and pass only parameters with requires_grad enabled to AdamW. After the third epoch, unfreeze the upper layers, rebuild the optimizer, and inspect each layer’s final trainable status. section 3.4.4

ml={0 for t<τunfreeze(l) 1 for t≥τunfreeze(l)}

Shows a time-dependent mask that switches a layer from frozen to trainable after its unfreezing threshold. section 3.4.2

How Building Specialized AI Systems covers it

How deep it goes
Descends from freezing rationale to gradient criteria, dynamic schedules, sublayer choices, and PyTorch implementation details.
Examples
Demonstrates BERT-base binary classification with six frozen encoder layers, partial attention freezing, AdamW, and epoch-three unfreezing.
What it assumes
Assumes pretrained-model fine-tuning and gradient-based training, while defining layer selection through concrete transformer examples.
What you can do after
Apply it via PyTorch and Transformers by toggling requires_grad, filtering optimizer parameters, scheduling unfreezing, and checking final status.
How it connects
Connects fine-tuning with gradient descent, backpropagation, optimizers, catastrophic forgetting, and parameter-efficient fine-tuning.

Also explained in

Compare how 5 books explain freeze →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.