Layer-wise fine-tuning adapts a pretrained neural network sequentially, training one layer or layer group while freezing every other layer. It offers more controlled specialization than updating the entire model simultaneously. section 3.3.2 section 3.3
A typical LIFT procedure follows this sequence:
- Freeze all layers except the layer closest to the output, which is usually the most task-specific. section 3.3.2
- Tune the active layer, freeze it again, and unfreeze the next layer down. section 3.3.2
- Continue until the desired depth has been updated; gradients flow only through the active layer at each stage. section 3.3.2
Section-by-section practice
LIFT resembles learning an instrument by rehearsing one section at a time: difficult passages are polished before the next section is addressed. section 3.3.2
