A classifier can decide whether a proposed CT-scan region is a nodule, but it cannot find candidate regions on its own. Segmentation helps with that earlier step by marking individual pixels. SAM is a pretrained, transformer-based model that generates segmentation masks, either automatically or with a prompt indicating what to select. section 15.0 section 15.1 section 15.3.1
Three components turn an image and an optional prompt into a mask:
- The image encoder extracts features from image patches using a Vision Transformer. section 15.4
- The prompt encoder represents points, boxes, or masks numerically. section 15.4
- The mask decoder combines image features and prompt information to predict regions. For an ambiguous prompt, SAM can return three masks of different sizes with scores. section 15.4
Automatic masks versus a pointed-at object
With pretrained weights, SamAutomaticMaskGenerator produced 79 masks for an example image. Each mask was an image-sized binary array: True marked pixels belonging to a selected region. When the goal is a particular region instead, SamPredictor accepts an image and a positive point prompt; a point on a dog’s nose yielded masks associated with that point. section 15.4.1 section 15.5
image_path = "data/p1ch2/bobby.jpg"
image = Image.open(image_path).convert("RGB")
input_points = [[(320, 260)]]
input_points = np.array([(320, 260)])
predictor.set_image(np.array(image))
masks, _, _ = predictor.predict(input_points, point_labels=np.array([1]))
The point-prompt code shows the practical sequence: set the image, then call predict with point coordinates and a positive point label. section 15.5
For CT scans, a reference point can guide SAM toward a candidate nodule mask on a 2D slice without first training it on that specific task. That is distinct from the later task-specific training step: the nodule segmentation model fine-tuned for that step is SegFormer, not SAM. section 15.3.1 section 15.9
