liveBook · Manning

Fine-tuning

As explained in Deep Learning with PyTorch, Second Edition

The process of adapting a pretrained model to a new task by continuing training on task-specific data.

Fine-tuning means continuing to train a pretrained model on data for a particular task. Rather than starting with random weights, you build on features the model has already learned and adapt them to a new dataset. The same idea applies to improving prelearned word embeddings while solving a text task. section 15.7.1 section 4.5.5

For CT scans, the goal is to mark possible lung nodules without asking someone to supply a point for every image. Segment Anything (SAM) can make a mask when given a CT slice and a point; those masks become training targets for a lighter SegFormer model that can segment slices without prompts. The resulting candidates can then go to a separate classifier. section 15.6.3 section 15.7.1 section 15.1

A practical fine-tuning workflow is:

  • Prepare paired inputs and targets: CT slices and their nodule masks. Keep the filenames linked through metadata, then create training and validation datasets and dataloaders. section 15.6.3 section 15.7.1
  • Load a pretrained segmentation model, set labels for background and nodule, and train it on the paired data. The SegFormer example uses AdamW. section 15.7.1
  • Check training and validation losses, save the parameters, and test the model on unseen CT scans by visualizing its masks. section 15.9

Starting from a pretrained SegFormer

The key step is loading a checkpoint rather than creating an untrained model: from transformers import SegformerForSemanticSegmentation model = SegformerForSemanticSegmentation.from_pretrained( "nvidia/mit-b0", num_labels=2, id2label={"0": "background", "1": "nodule"}, label2id={"background": "0", "nodule": "1"}, ) optimizer = torch.optim.AdamW(model.parameters(), lr=0.00006) Some segmentation-head weights are newly initialized, so loading the checkpoint is only the start: the model still needs task-specific training. section 15.7.1

mask_filepath = os.path.join(mask_folder, filename)
x, y = center_irc[2], center_irc[1]
input_points = [[x, y]]

predictor = SamPredictor(sam)
predictor.set_image(ct_image_array)
masks, _, _ = predictor.predict(point_coords=np.array(input_points),
point_labels=np.array([1]), multimask_output=False)

mask_image_array = masks[0]
mask_image = Image.fromarray(mask_image_array)
mask_image.save(mask_filepath)

See how a point supplied to SAM produces the mask saved alongside a CT image for fine-tuning. section 15.6.3

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes
Works through data preparation, checkpoint loading, an optimizer, and training results without detailing SegFormer internals.
Examples
Uses SAM-generated CT masks to fine-tune SegFormer and proposes fine-tuning a pretrained Vision Transformer on CIFAR-10.
What it assumes
Introduces the needed model and dataset setup; treats earlier CT classification work as helpful but not required.
What you can do after
Apply it via Hugging Face Transformers to train SegFormer on CT masks, monitor losses, and visualize predictions on unseen scans.
How it connects
Links pretrained models, semantic segmentation, dataloaders, AdamW, and a downstream classifier.
Cover of Deep Learning with PyTorch, Second Edition
liveBookmanning.com
Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

Compare how 37 books explain fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.