Image segmentation partitions an image into areas and assigns a semantic label to each pixel, producing a segmentation mask. Unlike image classification, which labels an entire image, segmentation preserves precise spatial information; unlike object detection, it does not stop at bounding boxes. section 11.1 section 11.5
Figure 11.1 The three main computer vision tasks: classification, segmentation, and detection
Compare classification, segmentation, and object detection by the kind of spatial information each prediction retains. section 11.1
A from-scratch model first compresses the image with strided Conv2D layers, then expands the feature map with Conv2DTranspose layers until the output matches the input dimensions. Strided convolutions retain location information better than 2 × 2 max pooling, whose single value loses the winning location within each window. section 11.3.2 section 11.5
Training and inspecting a segmentation model
A practical R/Keras workflow normalizes images, predicts three per-pixel classes with a softmax output, compiles with Adam and sparse categorical cross-entropy, tracks foreground IoU, saves the best checkpoint, and compares a predicted mask with its target mask. A pretrained Segment Anything model provides an alternative and accepts point or box prompts. section 11.3.2 section 11.5
Segmentation provides more information than detection but costs more computation and requires more expensive pixel-precise annotations. Use detection instead when boxes are sufficient, such as counting objects without needing their exact shapes. section 12.0
