A segmentation mask labels an image pixel by pixel. It has the same height and width as the image, but a single channel: each integer gives the class of the corresponding pixel. In the Oxford-IIIT Pets dataset, mask values mean foreground (1), background (2), or contour (3). A mask is therefore a more detailed target than one label for an entire image. section 11.3.1
Pair a pet photo with its mask
Each JPG pet photo has a same-named PNG mask in annotations/trimaps/. Load the photo with three channels and the mask with one, then resize both to 200 × 200. Subtract 1 from the mask values to turn classes 1–3 into 0–2. A batch then contains image tensors shaped (32, 200, 200, 3) and target masks shaped (32, 200, 200, 1). Displaying them side by side makes the pixel alignment easy to check. section 11.3.1
Listing 11.2 Preparing the dataset
library(tfdatasets, exclude = "shape")
img_size <- shape(200, 200)
tf_image_load <- function(path, target_size = NULL, ...) {
img <- path |>
tf$io$read_file() |>
tf$io$decode_image(..., expand_animations = FALSE)
if (!is.null(target_size))
img <- img |> tf$image$resize(target_size)
img
}
make_dataset <- function(image_paths) {
stopifnot(is.data.frame(image_paths),
names(image_paths) == c("input", "target"))
tensor_slices_dataset(image_paths) |>
dataset_map(function(example_paths) {
input_image <- example_paths$input |>
tf_image_load(channels = 3L, target_size = img_size) #1
target <- example_paths$target |>
tf_image_load(channels = 1L, target_size = img_size) #2
target <- tf$cast(target, "uint8") - 1L #3
list(input_image, target)
}) |>
dataset_cache() |> #4
dataset_shuffle(buffer_size = nrow(image_paths)) |> #5
dataset_batch(32)
}
num_val_samples <- 1000 #6
image_paths <- all_image_paths |>
dplyr::mutate(
use = ifelse(sample.int(n()) > num_val_samples, "train", "val") #7
) |>
tidyr::nest(.by = use) |>
tibble::deframe()
train_ds <- make_dataset(image_paths$train) #8
val_ds <- make_dataset(image_paths$val) #8
The dataset-preparation listing shows the essential pairing: load a three-channel input and a one-channel target at the same size, then shift the target’s class values down by one. section 11.3.1
To predict a mask, a segmentation model must preserve useful location information while reducing and then restoring spatial size. Strided convolutions compress the pet image into feature maps; transposed convolutions expand them. Its 200 × 200 × 3 output gives class predictions at every pixel, rather than the single prediction produced by an image classifier. Sparse categorical crossentropy trains against the integer target mask, while foreground intersection over union tracks overlap for the foreground class. section 11.3.2
Masks can also be produced without training a pet-specific model:
- Segment Anything accepts an image and a point inside an object or a box around it, then predicts candidate masks. Point prompts on fruit yield several possible masks, useful when the intended boundary is ambiguous. section 11.4.2 section 11.4.4
- A box around a mango can guide mask creation. Such prompted predictions can help annotate images for a segmentation dataset. section 11.4.5
Choose a mask when pixel-level shape matters. If only an object’s location is needed, a bounding box is usually cheaper: detection models typically run faster, and drawing boxes takes less labeling work than producing pixel-precise masks. A box can be computed from an existing object mask, but not all tasks need that extra detail. section 12.0
