Chapter 11: Image segmentation
The introduction positions image segmentation as another computer vision application beyond image classification and previews the chapter’s topics.
After this chapter you will be able to
- Three core vision tasks. Distinguish image classification, image segmentation, and object detection by their outputs: image labels, regions identified at pixel-level precision, and class-labeled bounding boxes.
- Pixel-level image segmentation. Explain that image segmentation assigns a class to each pixel, dividing an image into zones such as background and foreground or road, car, and sidewalk.
- Three segmentation types. Distinguish semantic, instance, and panoptic segmentation using an image with two cats: semantic segmentation gives both cats' pixels the generic cat category, instance segmentation separates the two cats, and panoptic segmentation supplies both category and instance labels.
- Pixel-Level Segmentation Masks. Explain how a segmentation mask labels an input image pixel by pixel, and identify the three mask values used in this dataset.
- Paired Dataset Preparation. Describe how the section pairs Oxford-IIIT Pets images with their masks and prepares them for training and validation.
- Image encoding into feature maps. Describe how the first half of the model compresses an image into smaller feature maps while giving each spatial location information about a larger part of the original image.
- Location preserving downsampling choice. Justify using strided convolutions rather than max pooling to downsample in this segmentation model.
- Upsampling to pixel predictions. Explain how the second half uses Conv2DTranspose layers to restore spatial dimensions before producing a class prediction for each pixel.
- Intersection over Union metric. Describe how Intersection over Union measures agreement between predicted and ground-truth segmentation masks.
- Class independent SAM segmentation. Explain that SAM is not restricted to predefined object classes: a user can indicate an example of the desired object and segment new objects without first fine-tuning the model.
- Pretrained segmentation model setup. Explain how KerasHub’s ImageSegmenter lets the reader instantiate SAM and download its weights without implementing the model from scratch.
- Prompted mask prediction. Describe the inputs and target in an (image, prompt, mask) training example, including the two prompt forms described here.
- Encoder and decoder roles. Trace how SAM turns an image and a prompt into possible predicted masks through its three high-level components.
- Training triple sampling. Describe how training triples are repeatedly sampled from SA-1B images and masks, including how point and box prompts are constructed.
- Aspect-preserving SAM input preparation. Explain how to make a nonsquare test image meet SAM’s 1024 × 1024 input requirement without distorting it.
- Labeled point prompts. Explain how a point prompt identifies an object for SAM to segment, and how additional foreground and background points can clarify an ambiguous prompt.
- Candidate masks and ambiguity. Describe the masks and quality scores SAM returns for a point prompt, and explain why alternative masks can be useful.
- Target box prompting. Describe how to prompt SAM to segment an object using a box that approximates its location, specified by its top-left and bottom-right corner coordinates.
What it builds on
- Training, validation, and test sets (chapter 5) · The distinction between training and validation sets.
- The convolution operation (chapter 8) · The convolution operation used in ConvNets.
- The max-pooling operation (chapter 8) · Max pooling as a way to downsample feature maps.
- Underfitting and overfitting (chapter 5) · The idea of overfitting.
- Using a pretrained model (chapter 8) · The earlier use of pretrained models for image classification.
- Feature extraction with a pretrained model (chapter 8) · The idea that an image model can extract a compact representation of an image.
Section by section
Introduction
The introduction positions image segmentation as another computer vision application beyond image classification and previews the chapter’s topics.
11.1 Computer vision tasks
The section distinguishes image classification, image segmentation, and object detection by what each produces from an image.
- Three core vision tasks: Distinguish image classification, image segmentation, and object detection by their outputs: image labels, regions identified at pixel-level precision, and class-labeled bounding boxes.
Why it matters: These output differences help you recognize which vision task fits an application. - Single-label and multilabel classification: Distinguish single-label classification from multilabel classification by whether the categories are mutually exclusive or an image can receive all applicable category tags.
Why it matters: Not every classification problem assumes that an image belongs to just one category.
Key terms: image classification, image segmentation, object detection, bounding boxes, single-label classification, multilabel classification, mutually exclusive
11.1.1 Types of image segmentation
The section defines image segmentation as pixel-level labeling and distinguishes semantic, instance, and panoptic segmentation.
- Pixel-level image segmentation: Explain that image segmentation assigns a class to each pixel, dividing an image into zones such as background and foreground or road, car, and sidewalk.
Why it matters: Pixel-level labels are the common foundation of all three segmentation types. - Three segmentation types: Distinguish semantic, instance, and panoptic segmentation using an image with two cats: semantic segmentation gives both cats' pixels the generic cat category, instance segmentation separates the two cats, and panoptic segmentation supplies both category and instance labels.
Why it matters: The distinctions tell you what information a segmentation result provides about objects in an image.
Key terms: image segmentation, pixel, class, semantic segmentation, instance segmentation, panoptic segmentation, semantic label, instance label
11.2 Training a segmentation model from scratch
Introduces a forthcoming semantic segmentation example using cat and dog images to distinguish the main subject from its background.
11.2.1 Downloading a segmentation dataset
This section introduces the Oxford-IIIT Pets segmentation dataset, explains its pixel-level masks, and prepares paired inputs and targets for training.
- Pixel-Level Segmentation Masks: Explain how a segmentation mask labels an input image pixel by pixel, and identify the three mask values used in this dataset.
Why it matters: The mask specifies the pixel-by-pixel target the segmentation model will learn to predict. - Paired Dataset Preparation: Describe how the section pairs Oxford-IIIT Pets images with their masks and prepares them for training and validation.
Why it matters: Keeping images paired with their masks gives the model the right target for each training input.
Key terms: segmentation mask, foreground, background, contour, Oxford-IIIT Pets dataset, images/, annotations/trimaps/, NumPy arrays, training, validation
11.2.2 Building and training the segmentation model
The section explains how a segmentation model compresses an image, upsamples features into a per-pixel mask, and evaluates the result.
- Image encoding into feature maps: Describe how the first half of the model compresses an image into smaller feature maps while giving each spatial location information about a larger part of the original image.
Why it matters: This compression explains why the model needs an upsampling half to produce a pixel-sized mask. - Location preserving downsampling choice: Justify using strided convolutions rather than max pooling to downsample in this segmentation model.
Why it matters: A per-pixel prediction depends on preserving where image features occur. - Upsampling to pixel predictions: Explain how the second half uses Conv2DTranspose layers to restore spatial dimensions before producing a class prediction for each pixel.
Why it matters: Restoring the image’s spatial dimensions makes a prediction at each pixel possible. - Intersection over Union metric: Describe how Intersection over Union measures agreement between predicted and ground-truth segmentation masks.
Why it matters: IoU gives a direct measure of how closely a predicted mask matches its target. - Validation loss and mask inspection: Describe what the training results show and how the section chooses and inspects a model for prediction.
Why it matters: The example uses validation performance, rather than simply the end of training, to select the model it demonstrates.
Key terms: Conv2D, feature maps, downsample, strides, max pooling, location information, per-pixel target masks, Conv2DTranspose, upsample, num_classes
11.3 Using a pretrained segmentation model
This section explains why a pretrained model is useful for image segmentation and introduces SAM’s ability to segment objects without a fixed class list or fine-tuning.
- Pretrained segmentation advantage: Explain why a pretrained model can help with image segmentation when training samples are scarce, and how SAM’s large training dataset gives it broad knowledge of objects in natural images.
Why it matters: It explains why the section turns to a pretrained segmentation model rather than relying only on training from scratch. - Class independent SAM segmentation: Explain that SAM is not restricted to predefined object classes: a user can indicate an example of the desired object and segment new objects without first fine-tuning the model.
Why it matters: This flexibility is what makes SAM useful beyond a fixed set of segmentation categories.
Key terms: pretrained model, Segment Anything Model, segmentation masks, SAM, predefined set of object classes, fine-tune
11.3.1 Downloading the Segment Anything Model
This section shows how to obtain a pretrained Segment Anything Model through KerasHub’s ImageSegmenter task and identifies the chosen preset.
- Pretrained segmentation model setup: Explain how KerasHub’s ImageSegmenter lets the reader instantiate SAM and download its weights without implementing the model from scratch.
Why it matters: This is how the chapter gets a ready-to-use segmentation model before testing it on images. - SAM preset identity and scale: Describe what the sam_huge_sa1b preset name indicates and state the model’s parameter count.
Why it matters: The preset name tells you which pretrained model is being used and signals its unusually large scale.
Key terms: KerasHub, ImageSegmenter, SAM, sam_huge_sa1b, SA-1B, 641 million parameters
11.3.2 How Segment Anything works
This section explains how Segment Anything’s segmentation data, prompted training examples, and model components work together to predict object masks.
- SA-1B dataset development: Explain how SA-1B and Segment Anything were developed together, with a partially trained model helping people label images.
Why it matters: The data-collection process helps explain how the model obtained such extensive segmentation training data. - Prompted mask prediction: Describe the inputs and target in an (image, prompt, mask) training example, including the two prompt forms described here.
Why it matters: The prompt specifies which object the predicted mask should cover. - Encoder and decoder roles: Trace how SAM turns an image and a prompt into possible predicted masks through its three high-level components.
Why it matters: This high-level path explains how the model combines what is in the image with the object a user selected. - Training triple sampling: Describe how training triples are repeatedly sampled from SA-1B images and masks, including how point and box prompts are constructed.
Why it matters: Sampling prompts from labeled masks supplies many examples of the task SAM is trained to perform.
Key terms: SA-1B, segmentation mask, semiautomated, (image, prompt, mask) triples, point prompt, box prompt, ground truth mask, image encoder, image embedding, prompt encoder
11.3.3 Preparing a test image
The section prepares a test image for SAM without distorting it and introduces utilities for displaying segmentation results.
- Aspect-preserving SAM input preparation: Explain how to make a nonsquare test image meet SAM’s 1024 × 1024 input requirement without distorting it.
Why it matters: This gives SAM the input size it expects while preserving the image’s proportions. - Shared figure display utilities: Describe why the image-display utilities take a Matplotlib axis object.
Why it matters: A shared figure makes it possible to show segmentation masks, points, or boxes alongside the image.
Key terms: SAM, aspect ratio, pad_to_aspect_ratio, keras.ops.image.resize(), Matplotlib axis object, ax, segmentation masks
11.3.4 Prompting the model with a target point
The section shows how to prompt SAM with labeled points and interpret its alternative segmentation masks.
- Labeled point prompts: Explain how a point prompt identifies an object for SAM to segment, and how additional foreground and background points can clarify an ambiguous prompt.
Why it matters: Point labels let you tell SAM both what to include and what to leave out. - Candidate masks and ambiguity: Describe the masks and quality scores SAM returns for a point prompt, and explain why alternative masks can be useful.
Why it matters: Looking beyond the first mask can reveal another plausible interpretation of your prompt.
Key terms: SAM, point prompt, foreground, background, masks, candidate masks, iou_pred, ambiguous prompts
11.3.5 Prompting the model with a target box
This section shows how to prompt SAM with a box around an object and notes its use for creating segmentation-mask datasets.
- Target box prompting: Describe how to prompt SAM to segment an object using a box that approximates its location, specified by its top-left and bottom-right corner coordinates.
Why it matters: A box gives you another way to indicate the object whose mask you want. - Segmentation dataset creation: Explain the potential use of SAM for quickly creating large datasets of images annotated with segmentation masks.
Why it matters: It connects prompting an individual object to a practical use for segmentation outputs.
Key terms: target points, boxes, top-left and bottom-right corners, SAM, segmentation masks
11.4 Summary
Where this is used later
- Single-stage vs. two-stage object detectors (chapter 12) · Builds on the basic meaning of object detection.
- (chapter intro) (chapter 12) · The earlier discussion of image segmentation provides context for segmentation masks.
- Implementing a VAE with Keras (chapter 17) · The earlier image segmentation example that also used strides for downsampling.
Concepts explained in this chapter
- segmentation maskAn image-shaped target annotation for segmentation where each pixel stores the class or region of the corresponding input pixel.
- SAMA pretrained segmentation model available through KerasHub for prompting-based segmentation tasks.
- image segmentationA vision task that assigns a label to each pixel or region of an image, providing denser localization than object detection.
- promptA short text description used to condition image generation, including both positive and negative forms in this section.
- classifierA supervised learning task that assigns text inputs to one of several classes.
- foundational modelA model trained beforehand on large-scale data and then reused for downstream tasks or analysis without starting from scratch.
- prompt engineeringThe practice of crafting prompts to elicit better performance from a large language model on a task.
- bounding boxA rectangle representation used to localize an object in an image, typically encoded by coordinates and size.
- KerasHubA Keras-hosted library providing pretrained models and high-level tasks such as Stable Diffusion and TextToImage.
- convolutional layerA two-dimensional convolution layer used here with a 1×1 kernel to project a residual to a matching shape.
- data annotationThe process of having people manually label or correct dataset examples.
- embeddingAn ordered numeric representation used here to encode source tokens and target tokens for attention computations.
- prompt encoderA model component that converts a prompt into an embedded vector representation.
- visual encoderA model component that converts images into vector representations suitable for a larger multimodal model.
- fine-tuningAdapting a pretrained model to a new task by continuing training, usually with a smaller learning rate or on additional data.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.