Image segmentation assigns a class to every pixel, partitioning an image into meaningful regions such as foreground and background, road, car, or sidewalk. Unlike object detection, which returns bounding boxes, segmentation provides pixel-level localization. section 11.1 section 11.1.1
Choose the segmentation flavor according to the detail you need:
- Semantic segmentation labels each pixel by category; separate objects sharing a category receive the same label. section 11.1.1
- Instance segmentation separates individual object instances, such as cat 1 and cat 2. section 11.1.1
- Panoptic segmentation combines semantic and instance labels, assigning both kinds of information to every pixel. section 11.1.1
A foreground mask for pet images
The Oxford-IIIT Pets dataset pairs 7,390 cat and dog photographs with same-sized grayscale masks. Each mask stores foreground, background, and contour as integer values 1, 2, and 3. A practical pipeline downloads the archives, matches JPG inputs with PNG masks, resizes images to 200×200, loads arrays into memory, and creates training and validation splits. section 11.2.1
A segmentation model can be built by compressing an image with strided Conv2D layers and expanding the feature map with corresponding Conv2DTranspose layers until it returns a mask matching the input size. A pretrained Segment Anything model offers image, text, point, and box prompting when training from scratch is unnecessary. section 11.4
Figure 11.2 Semantic segmentation vs. instance segmentation
Compare semantic and instance segmentation visually before choosing a labeling scheme. section 11.1.1
