Chapter 15: Using segmentation to find suspected nodules

Introduction

The introduction explains why the existing nodule classifier needs an automatic way to find candidates in CT scans and places that need within a multistage project design.

• Hand-annotated candidate dependency: Explain why the existing classifier does not yet provide an automatic nodule-finding pipeline: it classifies supplied CT scan candidates, but loading those candidates still depends on hand-annotated information.
  Why it matters: It clarifies the gap between classifying a suspected nodule and finding one in a CT scan.
• Multistage project design: Describe the project's separation of finding possible nodules from identifying them as nodules or non-nodules, and contrast this practitioner-style approach with the research tendency to demonstrate a single end-to-end model.
  Why it matters: It explains why the project addresses candidate finding and candidate classification as separate tasks.

Key terms: CT scan candidates, classifier, hand-annotated nodule candidate information, annotations.csv, candidates.csv, multiple steps, multistage project design, end-to-end

15.1 Utilizing a second model in our project

This section explains why the lung cancer detection project adds segmentation before classification and why it uses 2D CT slices.

• Segmentation before classification: Explain how segmentation flags CT voxels that might belong to nodules so the classifier has possible nodules to examine, and why some flagged regions will not be nodules.
  Why it matters: It connects the new segmentation step to the classifier developed in the previous chapters.
• Choice of 2D CT slices: Justify using 2D CT slices rather than 3D CT data for the segmentation work described here.
  Why it matters: It explains a practical choice that shapes how the segmentation results are presented.

Key terms: segmentation, voxels, classification, nodule, 2D slices, 3D, CT scan

15.2 Various types of segmentation

The section defines semantic segmentation for finding nodule candidates and contrasts it with instance segmentation and object detection.

• Semantic Segmentation Candidate Mask: Explain how semantic segmentation labels individual image pixels and how this project uses a binary label mask to mark nodule candidates for later classification.
  Why it matters: The mask helps identify candidate areas to feed into the classification network.
• Segmentation Alternatives And Tradeoffs: Distinguish semantic segmentation from instance segmentation and object detection, and explain why this project favors semantic segmentation.
  Why it matters: These distinctions explain why the project uses pixel labels rather than separate tumor identities or boxes.

Key terms: semantic segmentation, label mask, heatmap, nodule candidates, instance segmentation, object detection, bounding box

15.3 Semantic segmentation: Per-pixel classification

The section contrasts whole-image classification with segmentation, which preserves where objects appear in an image.

• Spatial segmentation outputs: Distinguish classification’s whole-image class predictions from segmentation’s image-like output that indicates where an object is. Explain why segmentation is useful when the locations of objects matter.
  Why it matters: Choosing between classification and segmentation depends on whether the task needs an object’s location, not just its presence.
• Classification funnel structure: Trace how convolutional classification models turn pixels into a small set of class predictions, and explain the effect of downsampling on the area considered by later detectors.
  Why it matters: The classification model’s funnel shape helps explain why its final answer does not show where the object is.

Key terms: classification, segmentation, image-like output, binary flags, convolution, downsampling, receptive field, feature detectors

15.3.1 The Segment Anything model (SAM)

Introduces Segment Anything (SAM) as a broadly applicable segmentation model and explains why its zero-shot capability makes it significant.

• SAM segmentation capabilities: Describe what SAM can produce from an image and the kinds of prompts it accepts.
  Why it matters: These capabilities explain why SAM is a candidate for segmenting nodules in CT scans.
• Zero-shot segmentation significance: Explain what zero-shot segmentation means for SAM and why the section presents it as a potential foundational model.
  Why it matters: Zero-shot segmentation is the key reason the section treats SAM as a step beyond task-specific segmentation models.

Key terms: Segment Anything model, SAM, segmentation masks, prompts, zero-shot segmentation, fine-tuning, foundational model, task-specific training data

15.4 SAM architecture

This section explains SAM’s main components and mask outputs, then highlights practical considerations when reusing an open-source model.

• SAM component roles: Trace how SAM’s image encoder, prompt encoder, and mask decoder work together to produce a segmentation mask.
  Why it matters: The three roles explain how an image and a prompt become a mask.
• Multiple masks for ambiguity: Explain why SAM can return multiple masks for one prompt and describe the outputs presented here.
  Why it matters: A single prompt need not correspond to just one plausible region.
• Open-source license limits: Explain why a project must check the licenses of open-source model code and pretrained weights.
  Why it matters: Finding usable model code also means checking what its authors permit you to do with it.
• Existing model reuse: Justify looking for existing models or reusable model parts when approaching a project.
  Why it matters: Knowing the available models can help you spot a useful starting point or reusable component.

Key terms: image encoder, prompt encoder, mask decoder, embeddings, segmentation mask, ambiguity, output masks, confidence score, whole, part

15.4.1 Trying out an off-the-shelf model for our project

This subsection walks through running pretrained Segment Anything on a sample image and interpreting its automatically generated masks.

• Pretrained automatic mask generation: Describe how the example uses an off-the-shelf Segment Anything model to generate masks for an image: install the library, load pretrained weights, and pass the image through SamAutomaticMaskGenerator.
  Why it matters: This shows how to try segmentation without first training a model.
• Generated masks and confidence scores: Explain what the example’s generated masks and scores represent, and what viewing one mask or all masks reveals.
  Why it matters: Reading the masks correctly lets you interpret what the model segmented and how confident it is.

Key terms: checkpoint, pretrained weights, SamAutomaticMaskGenerator, masks, binary arrays, scores tensor, confidence scores

15.5 Using the SAM model directly

The section shows how to use a point prompt with SAM to target a specific region, then cautions that results on natural images may not carry over to CT scans.

• Targeted Masks Instead of All Masks: Explain why the project moves from automatic mask generation to prompting SAM directly: it needs to single out suspected nodules rather than generate every mask in an image.
  Why it matters: Targeting a region makes segmentation useful for finding suspected nodules rather than merely listing image masks.
• Point Prompt Segmentation Flow: Trace the point-prompt example from a supplied location through SAM’s representations to its mask output, including why the example indexes the prompts with [0].
  Why it matters: This explains what the supplied point changes and how to interpret the example’s several masks.
• Natural Versus Medical Image Gap: Explain why the successful dog-image example does not establish that off-the-shelf SAM will work equally well on CT images, and state why the book nevertheless pursues this approach.
  Why it matters: It keeps the dog demonstration in perspective when the project turns to lung CT scans.

Key terms: SamAutomaticMaskGenerator, point prompt, nodules, prompt encoder, embedding representation, patches, masks, domain gap, natural images, medical images

15.6 Updating the dataset for segmentation

The section explains why the segmentation dataset must change shape even though its CT scans and annotations remain the same.

• Segmentation dataset dimensionality: Explain that the project still uses CT scans and their annotations, but the segmentation model needs data in a different form: the previous dataset produced 3D data, whereas this one must produce 2D data.
  Why it matters: This distinction explains why the dataset needs updating without suggesting that new source data is required.

Key terms: CT scans, annotation data, 3D data, 2D data

15.6.1 Working around SAM’s limitation on 2D data

The section explains how to use 2D CT slices with SAM despite the scans being 3D, and identifies the data and tradeoffs involved.

• Adapting CT scans for SAM: Explain why 3D CT scans must be split into 2D slices for SAM and how those slices are used to segment areas of interest for the classification model.
  Why it matters: Treating slices as images makes the CT scans usable with SAM for finding areas the classification model should examine.
• Slice geometry and tradeoffs: Describe what treating slices separately leaves out and why the section considers the approach workable for nodules.
  Why it matters: Slice spacing affects what a 2D image represents, even when processing slices individually is practical.
• Slice images and matching masks: State what paired data is needed to fine-tune SAM for this segmentation task.
  Why it matters: The masks identify the regions the model is being trained to segment.

Key terms: volumetric 3D images, 2D slices, SAM, areas of interest, slice thickness, slice spacings, nodules, fine-tune, CT scan slices, masks of interest

15.6.2 Building the segmentation dataset

This section explains how to prepare 2D CT slices and their point annotations for a segmentation dataset, including the need for pixel masks and cached slices.

• Point annotations versus pixel masks: Explain why the existing human-labeled data cannot directly serve as the desired segmentation output, and state how the section proposes obtaining masks.
  Why it matters: A point marking a nodule is not yet the pixel-by-pixel target needed for segmentation.
• CT slice extraction and caching: Trace how the adapted LunaDataset obtains a 2D CT slice for a candidate and why those slices are cached.
  Why it matters: Preparing and caching the right 2D slices makes the CT data usable for the segmentation dataset without repeatedly processing whole scans.

Key terms: annotated points, mask, base SAM, LunaDataset, series UID, xyz coordinates, CT slice, prepcache

15.6.3 Training a model to flag potential candidates

The section explains how to turn CT slices and SAM-generated masks into a dataset for fine-tuning a lighter segmentation model to flag potential nodule candidates.

• Segmentation and classification roles: Distinguish the segmentation model’s job of extracting potential candidates from the classification model’s job of deciding whether a candidate is a nodule, and explain why pixel-perfect masks are not the priority here.
  Why it matters: This distinction clarifies what the new model contributes to the existing detection project.
• SAM masks as training targets: Describe how pretrained SAM generates masks from 2D CT slices and corresponding point prompts, and explain why those masks are used to fine-tune SegFormer.
  Why it matters: SAM supplies labels without the expense of drawing every mask by hand, while SegFormer is intended to work without a prompt.
• Image and metadata dataset preparation: Trace how the section prepares paired CT images and masks from LunaDataset, records their metadata, and makes them accessible to a fine-tuning dataset.
  Why it matters: Saved pairs and their metadata let the training dataset locate the right image and mask together.

Key terms: segmentation, classification model, potential candidates, pixel-perfect masks, SAM, point, segmentation masks, ground truth, fine-tuning dataset, SegFormer

15.7 Updating our training for fine-tuning

This section introduces the upcoming changes to the training setup for fine-tuning the new model.

15.7.1 How to fine-tune a model

Explains why the project fine-tunes a pretrained segmentation model and identifies the model and data used to set it up.

• Prompt-free nodule segmentation: Explain why the project needs a model that segments nodules without a point prompt: it must find areas in unseen CT images to pass to the existing classification model.
  Why it matters: This explains why prompted segmentation alone does not meet the project's goal.
• Pretrained model adaptation: Explain fine-tuning as continuing to train a pretrained model on CT scan slices and masks so it adapts its previously learned features to this segmentation task.
  Why it matters: It clarifies why training can begin with an existing model rather than from scratch.
• SegFormer model setup: Describe the setup named here: use FineTuningDataset for images and masks, a dataloader to iterate over them, and the lightweight nvidia/mit-b0 SegFormerForSemanticSegmentation model. Explain why a message about some uninitialized checkpoint weights is expected.
  Why it matters: These choices connect the fine-tuning idea to the data and model used in the project.

Key terms: point prompt, classification model, nodules of interest, fine-tuning, pretrained model, CT scan slices, masks, FineTuningDataset, dataloader, SegFormerForSemanticSegmentation

15.7.2 Using the AdamW optimizer

This section explains how Adam adjusts learning rates and why the project uses AdamW for SegFormer fine-tuning.

• Adam learning rate adaptation: Explain how Adam differs from SGD in its treatment of learning rates and why its default learning rate is often sufficient.
  Why it matters: This explains why Adam is often straightforward to use when starting model training.
• AdamW selection rationale: Describe what AdamW adds to Adam and why this project uses it, while distinguishing that choice from the general case for starting with Adam.
  Why it matters: It connects the optimizer choice to the SegFormer reference rather than treating AdamW as universally best.

Key terms: Adam optimizer, SGD, learning rate, parameter, AdamW, Adam, weight decay, SegFormer, hyperparameters

15.7.3 Designing our training loop

The section explains how to preprocess CT images and masks for Segformer, train and validate the model, and inspect its losses in TensorBoard.

• Segformer input preprocessing: Explain why CT images and masks are passed through SegformerImageProcessor before they reach the model.
  Why it matters: The model needs its inputs prepared in the format it expects.
• Epoch training and loss averaging: Trace the training loop through batches and epochs, including how it reports training loss.
  Why it matters: It connects each training batch to the reported loss for an epoch.
• Validation without gradient computation: Distinguish validation from training in this loop and explain why its settings matter.
  Why it matters: Validation checks the model after each epoch without the gradient work used for training.
• Interpreting TensorBoard loss trends: Describe what the plotted training and validation losses suggest about the model.
  Why it matters: Seeing both loss curves fall gives a useful, though tentative, signal about training and generalization.

Key terms: CT scan images and masks, SegformerImageProcessor, Segformer, data loader, loss, model weights, epoch, average loss, validation set, eval()

15.7.4 Saving our model

The section explains why to save model parameters rather than the whole model, what a state dictionary contains, and how saved parameters can be loaded for later use.

• Parameter-only saving flexibility: Explain why the section favors saving model parameters instead of pickling the entire model instance.
  Why it matters: A parameter-only file gives you more options for reusing or adapting a trained model.
• State dictionary and restoration: Describe what model.state_dict() provides and the basic sequence for using saved parameters again.
  Why it matters: The state dictionary is the saved data that lets a newly instantiated model use its learned parameters.
• Additional checkpoint information: Describe what other information torch.save can store alongside model parameters and why it can be useful.
  Why it matters: A saved file can preserve useful training context as well as model parameters.

Key terms: torch.save, parameters, pickle, model.state_dict(), state dictionary, weights, biases, torch.load, model.to(device), optimizer state

15.8 Inference and results

This section shows how to run the trained segmentation model on CT slices and inspect its predicted nodule masks alongside the images and ground-truth masks.

• Inference and mask inspection: Describe how CT scan slices are passed to the trained model to produce nodule masks, and how the predictions are visualized with the images and ground-truth masks to assess performance.
  Why it matters: Seeing the predicted mask next to the scan and ground truth makes the model’s segmentation result inspectable.

Key terms: inference, CT scan slices, masks, ground truth mask

15.9 Conclusion

The chapter concludes by summarizing how prompted SAM masks, a paired image-and-mask dataset, and a fine-tuned SegFormer model were used to segment suspected nodules in CT scans.

• Prompted SAM nodule masks: Explain how the chapter used SAM to generate a mask for a candidate nodule from a CT image and a reference point.
  Why it matters: This identifies the role of the prompted model in the chapter’s segmentation work.
• Paired segmentation dataset: Describe the dataset created for segmentation and its purpose in the chapter.
  Why it matters: The image–mask pairs connect mask generation to training a segmentation model.
• SegFormer training and inference: Trace the high-level use of SegFormer from fine-tuning through evaluation on unseen CT scans.
  Why it matters: This is the chapter’s end-to-end result: a reusable model that produces nodule masks for scans it has not seen.

Key terms: Segment Anything model (SAM), segmentation masks, reference point, images, corresponding masks, dataset, SegFormer, fine-tuned, training and validation losses, TensorBoard

15.10 Exercises

15.11 Summary