Chapter 12: Object detection
This introduction defines object detection through bounding boxes and explains when it is preferable to image segmentation.
After this chapter you will be able to
- Bounding boxes and uses. Explain that object detection locates objects of interest by drawing bounding boxes around them, and describe how those locations can be used to count, track, or crop objects.
- Detection versus segmentation tradeoff. Distinguish the information supplied by segmentation from detection, and justify choosing detection when pixel-level information is unnecessary.
- Object detector architecture families. Distinguish two-stage R-CNN detectors from single-stage detectors by saying that two-stage detectors first extract region proposals, while RetinaNet and YOLO are examples of single-stage detectors.
- Two-stage R-CNN detection. Trace how an R-CNN goes from an image to detected objects: it proposes object-like boxes, classifies each proposal, discards low-scoring ones, and refines the remaining boxes.
- R-CNN computational tradeoff. Explain why two-stage R-CNN is computationally expensive and what that implies for real-time or resource-constrained use.
- Joint box and label prediction. Explain that a single-stage detector uses one deep learning model to predict bounding box coordinates together with object labels, and name YOLO, RetinaNet, or SSD as an example.
- Original YOLO model choice. Explain that building an object detector involves substantial code for manipulating bounding boxes and predicted output, rather than unusual theoretical complexity, and that the example uses the original YOLO model because it is simpler to work with.
- COCO annotations for detection. Describe what the COCO dataset provides and which annotations this example uses.
- Image and box association. Explain how the separately downloaded COCO images and metadata are brought together.
- Normalized bounding box coordinates. Explain the original COCO box coordinates and why the section rescales them.
- Grid predictions and confidence. Explain how this single-stage YOLO model predicts a bounding box and class label at each image-grid location, along with a confidence score that should be near zero where there is no object.
- Prediction head and global context. Trace how the detection head turns features into per-grid box, confidence, and class outputs, and explain why flattening lets each grid detector use features from the whole image.
- Grid responsibility and box coordinates. Explain which grid detector is responsible for a box and how the model represents that box's position and size.
- Class and box target arrays. Describe the two grid-aligned target arrays and how class labels, box coordinates, and confidence values are placed in them.
- Classification and box losses. Explain how training handles YOLO's predicted grid labels and boxes: use sparse categorical crossentropy for classification, and compute box-parameter error only for grid cells containing labeled boxes.
- IoU as confidence target. Explain why an object's target box confidence uses the IoU between its predicted and actual boxes, while an empty cell's target confidence is zero.
- Feature pyramid scale handling. Explain why the final ConvNet features used in the earlier YOLO model struggle with small objects, and how RetinaNet’s feature pyramid combines features from different depths to address this.
What it builds on
- Types of image segmentation (chapter 11) · The earlier discussion of image segmentation provides context for segmentation masks.
- Introduction to ConvNets (chapter 8) · Assumes familiarity with ConvNets as image-processing models.
- Feature extraction with a pretrained model (chapter 8) · Using a pretrained model to extract image features.
Section by section
Introduction
This introduction defines object detection through bounding boxes and explains when it is preferable to image segmentation.
- Bounding boxes and uses: Explain that object detection locates objects of interest by drawing bounding boxes around them, and describe how those locations can be used to count, track, or crop objects.
Why it matters: Knowing an object's location makes detection useful for tasks that image-level classification cannot handle alone. - Detection versus segmentation tradeoff: Distinguish the information supplied by segmentation from detection, and justify choosing detection when pixel-level information is unnecessary.
Why it matters: The extra detail of segmentation is not worth its computation and labeling costs when boxes are sufficient.
Key terms: object detection, bounding boxes, counting, tracking, cropping, segmentation mask, pixel-level information, pixel-precise masks
12.1 Single-stage vs. two-stage object detectors
Introduces the two broad families of object detectors and identifies region proposals as the first step of two-stage detectors.
- Object detector architecture families: Distinguish two-stage R-CNN detectors from single-stage detectors by saying that two-stage detectors first extract region proposals, while RetinaNet and YOLO are examples of single-stage detectors.
Why it matters: This distinction provides a starting point for comparing the detector architectures discussed next.
Key terms: two-stage detectors, region proposals, Region-based Convolutional Neural Networks (R-CNN), single-stage detectors, RetinaNet, You Only Look Once
12.1.1 Two-stage R-CNN detectors
This section explains how two-stage R-CNN detectors turn proposed image regions into object detections, how proposal generation changed over time, and why the approach can be too costly for some applications.
- Two-stage R-CNN detection: Trace how an R-CNN goes from an image to detected objects: it proposes object-like boxes, classifies each proposal, discards low-scoring ones, and refines the remaining boxes.
Why it matters: The two separate stages explain both what R-CNN outputs and where much of its processing work comes from. - Changing proposal generation methods: Distinguish early R-CNN proposal generation with Selective Search from the learned proposal generation used in later versions such as Faster-R-CNN.
Why it matters: Not every version of R-CNN generates its candidate boxes in the same way. - R-CNN computational tradeoff: Explain why two-stage R-CNN is computationally expensive and what that implies for real-time or resource-constrained use.
Why it matters: The cost of classifying so many proposals motivates considering a more efficient detector.
Key terms: R-CNN, region proposals, bounding boxes, ConvNet, class-presence score, Selective Search, heuristic, Faster-R-CNN, Region Proposal Network, two-stage approach
12.1.2 Single-stage detectors
This section defines single-stage object detectors and explains their speed–accuracy tradeoff relative to two-stage detectors.
- Joint box and label prediction: Explain that a single-stage detector uses one deep learning model to predict bounding box coordinates together with object labels, and name YOLO, RetinaNet, or SSD as an example.
Why it matters: This distinguishes the detector architecture used by the YOLO model built next. - Speed and accuracy tradeoff: Describe why single-stage detectors, particularly recent YOLO models, suit real-time applications while acknowledging their possible accuracy tradeoff against two-stage detectors.
Why it matters: The tradeoff explains YOLO’s appeal when detection must happen in real time.
Key terms: single-stage detector, bounding box coordinates, labels, YOLO, RetinaNet, Single Shot MultiBox Detectors (SSD), single-stage detectors, two-stage counterparts, real-time applications
12.2 Training a YOLO model from scratch
The section explains why the from-scratch example uses the original YOLO model: object detection requires substantial implementation work, and that version is simpler to work with.
- Original YOLO model choice: Explain that building an object detector involves substantial code for manipulating bounding boxes and predicted output, rather than unusual theoretical complexity, and that the example uses the original YOLO model because it is simpler to work with.
Why it matters: This explains the practical scope of the from-scratch example.
Key terms: object detector, bounding boxes, predicted output, original YOLO model
12.2.1 Downloading the COCO dataset
This section prepares COCO images and bounding-box annotations for a runnable YOLO training example.
- COCO annotations for detection: Describe what the COCO dataset provides and which annotations this example uses.
Why it matters: It identifies the annotations the detector will learn from. - Image and box association: Explain how the separately downloaded COCO images and metadata are brought together.
Why it matters: Matching boxes to the right photos is necessary before the data can be used for training. - Normalized bounding box coordinates: Explain the original COCO box coordinates and why the section rescales them.
Why it matters: A common coordinate scale avoids having to account for each image’s dimensions when handling boxes. - Training subset and shuffle: Justify limiting the images by box count and shuffling the remaining data.
Why it matters: These choices keep the example practical without preserving an unsuitable object-type ordering. - Labeled box visualization: Describe how the section makes an annotated example image readable.
Why it matters: Drawing the boxes makes it easier to inspect what an image’s annotations represent.
Key terms: COCO dataset, object labels, bounding box annotations, segmentation masks, image metadata, JSON file, ID, collate, bounding box, x, y, width, height
12.2.2 Creating a YOLO model
This section builds a simplified YOLO detector by adding grid-based box and class predictions to a pretrained ConvNet backbone.
- Grid predictions and confidence: Explain how this single-stage YOLO model predicts a bounding box and class label at each image-grid location, along with a confidence score that should be near zero where there is no object.
Why it matters: The grid and confidence score explain what the detector predicts even where no object appears. - Backbone features and downsampling: Describe why the model loads a pretrained ResNet backbone and downsamples its feature maps before the dense prediction layers.
Why it matters: The backbone provides useful image features, while downsampling makes them practical to feed into the prediction head. - Prediction head and global context: Trace how the detection head turns features into per-grid box, confidence, and class outputs, and explain why flattening lets each grid detector use features from the whole image.
Why it matters: This explains both the model's output structure and how a grid detector can use context beyond its own cell.
Key terms: single stage detector, grid, bounding box, class label, confidence score, ConvNet backbone, ResNet, pretrained backbone, strided conv layer, feature maps
12.2.3 Readying the COCO data for the YOLO model
This section converts COCO annotations into grid-aligned YOLO targets and prepares images for batched training.
- Grid responsibility and box coordinates: Explain which grid detector is responsible for a box and how the model represents that box's position and size.
Why it matters: The coordinate convention determines how training boxes must be aligned with the model's prediction grid. - Class and box target arrays: Describe the two grid-aligned target arrays and how class labels, box coordinates, and confidence values are placed in them.
Why it matters: These arrays provide the grid-shaped labels against which the YOLO model can train. - Streaming image data preparation: Explain how the image dataset is prepared for training and why images are loaded in batches rather than all at once.
Why it matters: Streaming makes the large dataset practical to train on without loading every image into memory.
Key terms: grid detector, grid cell, x, y, w, h, class probability map, boxes, confidence, grid
12.2.4 Training the YOLO model
This section defines the losses used to train a YOLO detector and interprets the limited results of a short training run.
- Classification and box losses: Explain how training handles YOLO's predicted grid labels and boxes: use sparse categorical crossentropy for classification, and compute box-parameter error only for grid cells containing labeled boxes.
Why it matters: It explains why training the detector calls for more than a single classification loss. - IoU as confidence target: Explain why an object's target box confidence uses the IoU between its predicted and actual boxes, while an empty cell's target confidence is zero.
Why it matters: It connects a detector's confidence score to the quality of its predicted location, not merely to whether an object exists. - Box loss scaling choices: Justify the section's loss adjustments: emphasize box placement, reduce the influence of empty-cell confidence errors, and transform widths and heights so large boxes do not dominate.
Why it matters: These adjustments prevent abundant empty cells and large boxes from distorting the training objective. - Short run detection results: Describe what the section concludes from the trained model's predictions and why the example does not yet produce an accurate detector.
Why it matters: It keeps the example's early predictions from being mistaken for a fully trained object detector.
Key terms: predicted boxes, predicted grid labels, sparse_categorical_crossentropy, sum-squared error, box confidence, Intersection over Union (IoU), target confidence value, box placement loss, confidence loss, square root
12.3 Using a pretrained RetinaNet detector
The section explains how RetinaNet’s feature pyramid helps detect objects at different scales, then uses a pretrained detector on a painting to illustrate its predictions and generalization.
- Feature pyramid scale handling: Explain why the final ConvNet features used in the earlier YOLO model struggle with small objects, and how RetinaNet’s feature pyramid combines features from different depths to address this.
Why it matters: This explains RetinaNet’s main architectural difference from the earlier YOLO model. - Pretrained RetinaNet predictions: Describe how the section runs pretrained RetinaNet inference and what its prediction contains, including why it selects the rel_xywh bounding-box format.
Why it matters: The box format and prediction fields explain how the pretrained detector’s results are displayed. - Painting generalization contrast: Explain the section’s account of why the single-stage RetinaNet detector handles the pointillist painting despite not being trained on that style, in contrast to a two-stage detector.
Why it matters: The painting example connects detector design to robustness on an unfamiliar visual style.
Key terms: feature pyramid network, Conv2DTranspose, lateral connections, feature maps, KerasHub, ObjectDetector, rel_xywh, bounding boxes, confidences, labels
12.4 Summary
Concepts explained in this chapter
- generationThe process of producing new text token by token from a trained language model, often by feeding previous outputs back as inputs.
- single-stage detectorAn object-detection architecture that predicts bounding boxes and class labels in one pass with a single model, rather than using separate proposal and classification stages.
- gridA grid is a regular arrangement of positions or cells organized by rows and columns or by fixed neighboring relationships.
- confidenceThe model's estimated certainty for a predicted detection, often represented as a score or probability.
- bounding boxA rectangle representation used to localize an object in an image, typically encoded by coordinates and size.
- YOLOA family of single-stage object-detection models that predict bounding boxes and classes directly from an image in one forward pass.
- bounding box predictionThe model output that estimates object-localizing rectangle coordinates for an image region.
- preprocessingThe transformation of raw data into a form suitable for input to a model.
- feature pyramid networkAn architecture that merges multi-scale feature maps from different depths to support detection of objects at multiple sizes.
- objective functionA function that compares predictions with targets and returns a scalar indicating how wrong the model is for a given task.
- loss scalingA technique that multiplies the loss by a scalar factor to increase gradient magnitudes and reduce underflow in low-precision training.
- trainingThe phase during which a model’s parameters are optimized from data before deployment for prediction.
- foundational modelA model trained beforehand on large-scale data and then reused for downstream tasks or analysis without starting from scratch.
- shuffleTo randomly reorder data before splitting it so that contiguous ordering does not bias the partition.
- prefetchA data-input optimization that overlaps data preparation with model execution by preparing upcoming batches in advance.
- tf.data APITensorFlow's input पाइപ്പ്-line API for building efficient data preprocessing and loading pipelines.
- KerasHubA Keras-hosted library providing pretrained models and high-level tasks such as Stable Diffusion and TextToImage.
- SAMA pretrained segmentation model available through KerasHub for prompting-based segmentation tasks.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.