Chapter 2: Pretrained networks
The introduction explains why deep learning has had a major impact on computer vision and why pretrained models are useful starting points.
After this chapter you will be able to
- Pretrained model behavior and value. Describe a pretrained neural network as an input-output system whose behavior reflects its architecture and training examples or desired output properties. Explain why an off-the-shelf model can jump-start a project.
- Pretrained Models and Weights. Explain that pretrained models can be obtained with weights learned from reference datasets, allowing image recognition without training a model from scratch.
- Image Classification Inference Flow. Trace how an RGB image is represented for the pretrained network and how its output becomes a predicted label.
- TorchVision model access and names. Explain that TorchVision provides computer-vision models in torchvision.models, and distinguish the model classes from the lowercase functions that instantiate them.
- AlexNet image processing path. Trace, at the level shown here, how an input image becomes AlexNet’s output probabilities, and describe what training determines about its filters.
- Forward passes and trained weights. Explain what calling an AlexNet instance does and why an instance without trained weights cannot yet give useful predictions. Distinguish the architecture class from the convenient model functions that can load pretrained weights.
- Vision Transformer classification performance. Explain that Vision Transformers use self-attention to capture relationships within images and have achieved lower ImageNet classification error than AlexNet.
- Model summary hierarchy. Explain what the printed model summary shows: modules are neural-network operations, nested modules appear as an indented hierarchy, and the network ends by producing class scores.
- Image preprocessing for inference. Describe how the section prepares an image for the callable vit model and why that preparation matters. Include resizing, center cropping, conversion to a tensor, and RGB normalization to match training.
- Inference and evaluation mode. Explain that running a trained model on new data is inference and that the network must be put in eval mode first.
- Scores to predicted label. Trace how the classifier’s output scores become a predicted ImageNet label.
- Limits of image predictions. Explain why a plausible-looking prediction, even one made with high confidence, may be wrong or reflect training-data biases.
- Inpainting mask regions. Explain what black and white pixels in an inpainting mask tell a pretrained diffusion model to do.
- Gradual noising and denoising. Explain that a diffusion model learns to reverse a process that gradually adds noise to images, removing noise step by step until a coherent picture emerges.
- Mask controlled local editing. Explain how a mask confines inpainting to its white region, with original content restored outside it during each step; contrast this with omitting the mask, which permits re-rendering the whole frame.
- Pretrained inpainting workflow. Describe how the example runs the diffusers inpaint pipeline: it supplies a horse image, a separately created mask, and a text prompt, then retrieves the generated image from the output.
- Learned weights behind image editing. Explain why the network can produce a zebra-like edit without zebra-specific source code or directly supervised horse-to-zebra examples.
- Model zoos and access. Explain what a model zoo contains and why a uniform interface for accessing pretrained models is useful.
- Hugging Face text generation. Describe what the section's Hugging Face GPT-2 example demonstrates and how the generated completion relates to its input.
- BLIP image text learning. Explain that BLIP is a multimodal model trained on images paired with sentence descriptions. Describe, at a high level, how comparing image and caption embeddings and generating caption guesses helps it learn their relationship.
- Caption generation pathway. Trace how BLIP generates a caption at inference time: an image encoder produces a descriptive numerical representation of the scene, and a text decoder turns that representation into a coherent sentence.
- BLIP captions of generated images. Describe how BLIP captions the horse image and still identifies the zebra in a generated image, despite the zebra appearing in an unlikely setting.
- Learned rather than hardcoded captions. Explain that the caption and its wording emerge from patterns in images and descriptions, not from horse- or zebra-specific rules or a handcoded grammar. State that the model involves multiple networks feeding into one another.
- Value of pretrained networks. Explain why a pretrained network can be a useful starting point for a new task, particularly when there are relatively few data points.
Section by section
Introduction
The introduction explains why deep learning has had a major impact on computer vision and why pretrained models are useful starting points.
- Computer vision advances: Explain why deep learning had a major impact on computer vision, connecting demand for image interpretation with large datasets, convolutional layers, and fast GPU computation.
Why it matters: These conditions help explain why image models are prominent examples of deep learning. - Pretrained model behavior and value: Describe a pretrained neural network as an input-output system whose behavior reflects its architecture and training examples or desired output properties. Explain why an off-the-shelf model can jump-start a project.
Why it matters: It explains what you are reusing when you run a model that someone else has trained.
Key terms: computer vision, convolutional layers, GPUs, pretrained neural network, architecture, off-the-shelf model, weights
2.1 A pretrained network that recognizes the subject of an image
The section introduces pretrained image classifiers, the ImageNet labels they use, and how an image becomes a ranked prediction.
- Pretrained Models and Weights: Explain that pretrained models can be obtained with weights learned from reference datasets, allowing image recognition without training a model from scratch.
Why it matters: This explains why you can try image recognition before learning to train a network. - ImageNet Classes and Predictions: Describe how ImageNet images are labeled and what the ILSVRC classification task asks an algorithm to predict.
Why it matters: The training labels explain what the classifier’s predicted categories refer to. - Image Classification Inference Flow: Trace how an RGB image is represented for the pretrained network and how its output becomes a predicted label.
Why it matters: This gives you the basic path from your own image to the label the model predicts.
Key terms: pretrained models, preoptimized weights, torchvision, Hugging Face, ImageNet, ILSVRC, class, label, top five predictions, preprocessed
2.1.1 Obtaining a pretrained network for image recognition
This section introduces TorchVision as a source of image-recognition models and explains how its model classes and convenience functions are named.
- TorchVision model access and names: Explain that TorchVision provides computer-vision models in torchvision.models, and distinguish the model classes from the lowercase functions that instantiate them.
Why it matters: This distinction helps you recognize what you are selecting when you obtain a model from TorchVision.
Key terms: TorchVision, torchvision.models, architecture, convenience functions
2.1.2 AlexNet
The section introduces AlexNet’s historical significance, sketches how an image passes through it, and explains why running it usefully requires trained weights.
- AlexNet's historical impact: Explain why AlexNet’s 2012 ILSVRC result changed how the computer-vision community viewed deep learning.
Why it matters: This explains why the book uses AlexNet as an early example of an image-recognition network. - AlexNet image processing path: Trace, at the level shown here, how an input image becomes AlexNet’s output probabilities, and describe what training determines about its filters.
Why it matters: This gives you a useful picture of what AlexNet does without requiring the details of its architecture. - Forward passes and trained weights: Explain what calling an AlexNet instance does and why an instance without trained weights cannot yet give useful predictions. Distinguish the architecture class from the convenient model functions that can load pretrained weights.
Why it matters: It clarifies why creating a runnable network is not the same as obtaining one that can recognize images.
Key terms: AlexNet, ILSVRC, top-five test error rate, filter, intermediate representations, 4,096-element 1D vector, output probabilities, output class, forward pass, weights
2.1.3 The Vision Transformer
This section introduces Vision Transformers for image classification, compares their ImageNet results with AlexNet, and identifies how to obtain a pretrained ViT in PyTorch.
- Vision Transformer classification performance: Explain that Vision Transformers use self-attention to capture relationships within images and have achieved lower ImageNet classification error than AlexNet.
Why it matters: This explains why a pretrained ViT is worth trying for image recognition. - Pretrained ViT selection: Describe using PyTorch’s vit_b_16 function with the specified ImageNet-trained weights to obtain a ViT for image classification.
Why it matters: The specified weights let the example use an already-trained image classifier.
Key terms: vision transformers, ViT, self-attention, top-five error rates, ImageNet, AlexNet, vit_b_16, ViT_B_16_Weights.IMAGENET1K_V1, weights
2.1.4 Ready, set, almost run
This section interprets a printed Vision Transformer model summary and prepares an image in the format the pretrained model expects.
- Model summary hierarchy: Explain what the printed model summary shows: modules are neural-network operations, nested modules appear as an indented hierarchy, and the network ends by producing class scores.
Why it matters: The summary becomes easier to read when you recognize its nested operations rather than treating it as an undifferentiated block of text. - Image preprocessing for inference: Describe how the section prepares an image for the callable vit model and why that preparation matters. Include resizing, center cropping, conversion to a tensor, and RGB normalization to match training.
Why it matters: Correct preprocessing lets the pretrained model interpret a new image in the way it was trained to.
Key terms: modules, layers, submodules, EncoderBlock, fc, out_features, vit, torchvision, transforms, preprocess
2.1.5 Run!
This section runs a pretrained image classifier and shows how to interpret its predictions while recognizing their limits.
- Inference and evaluation mode: Explain that running a trained model on new data is inference and that the network must be put in eval mode first.
Why it matters: Putting the model in the right mode is part of getting a usable prediction. - Scores to predicted label: Trace how the classifier’s output scores become a predicted ImageNet label.
Why it matters: A score only becomes a readable prediction when it is matched to the right class label. - Confidence and alternative predictions: Describe how the section obtains a confidence-like value and inspects predictions beyond the top one.
Why it matters: Looking beyond the winning label shows both its relative strength and the model’s other candidates. - Limits of image predictions: Explain why a plausible-looking prediction, even one made with high confidence, may be wrong or reflect training-data biases.
Why it matters: A confident class label is not a guarantee that the model has understood the image as a person would.
Key terms: inference, eval mode, batch normalization, dropout, ImageNet class, out tensor, max, index, .item(), labels
2.2 Generating and editing images
This section introduces diffusion-based inpainting as a way to make localized image edits and explains how a mask separates protected areas from editable ones.
- Inpainting mask regions: Explain what black and white pixels in an inpainting mask tell a pretrained diffusion model to do.
Why it matters: The mask is what lets an edit target one part of an image while preserving the rest. - Diffusion inpainting purpose: Describe what diffusion-based inpainting does and the broad characteristics of the approach stated here.
Why it matters: Inpainting provides a way to replace or edit a specific image area without training a new model.
Key terms: mask, black pixels, white pixels, protected areas, primed region, diffusion-based inpainting, text-guided, mask-aware, iterative refinement
2.2.1 The inpainting process
Explains how diffusion-based inpainting uses a prompt, an optional input image, and a mask to edit selected parts of a picture.
- Gradual noising and denoising: Explain that a diffusion model learns to reverse a process that gradually adds noise to images, removing noise step by step until a coherent picture emerges.
Why it matters: This explains what the model is doing as an edited image takes shape. - Prompt and image guidance: Distinguish starting from pure noise to generate a new image from starting with an existing picture to re-render it; describe how the prompt guides the result and the existing image anchors its layout.
Why it matters: The starting point determines whether the model makes a new picture or edits an existing composition. - Mask controlled local editing: Explain how a mask confines inpainting to its white region, with original content restored outside it during each step; contrast this with omitting the mask, which permits re-rendering the whole frame.
Why it matters: The mask is what lets an edit change the subject without repainting the surrounding scene.
Key terms: diffusion model, noising, denoising, prompt, input image, pure noise, image-to-image generation, mask, inpainting, masked region
2.2.2 A network that turns horses into zebras
This section uses a pretrained inpainting pipeline to turn a horse image into a zebra-like image, then explains what the example shows about learned weights and generative images.
- Pretrained inpainting workflow: Describe how the example runs the diffusers inpaint pipeline: it supplies a horse image, a separately created mask, and a text prompt, then retrieves the generated image from the output.
Why it matters: It shows how a complex pretrained image editor can be used through a simple interface. - Learned weights behind image editing: Explain why the network can produce a zebra-like edit without zebra-specific source code or directly supervised horse-to-zebra examples.
Why it matters: It distinguishes what the model learned from instructions a programmer explicitly wrote. - Generative image implications: Describe the broader possibilities and concerns the section draws from this image-editing example, including creative applications and convincing fake imagery.
Why it matters: The example has implications beyond a playful horse-to-zebra edit.
Key terms: diffusers, inpaint pipeline, mask, text prompt, weights, direct supervision, distant supervision, generative models, face-swapping, deep fakes
2.3 Model Zoo: Hugging Face
The section introduces model zoos through Hugging Face and uses pretrained GPT-2 text generation to show how readily a published model can be applied.
- Model zoos and access: Explain what a model zoo contains and why a uniform interface for accessing pretrained models is useful.
Why it matters: This explains what repositories such as Hugging Face offer when you want to use an existing model. - Hugging Face text generation: Describe what the section's Hugging Face GPT-2 example demonstrates and how the generated completion relates to its input.
Why it matters: The example connects a model repository to a concrete use of an already-trained model.
Key terms: model zoo, pretrained models, uniform interface, Hugging Face, transformers, GPT2, text generation, sampling strategy
2.4 A pretrained network that describes scenes
This section introduces how BLIP learns from paired images and captions and how its image encoder and text decoder generate a caption for a new image.
- BLIP image text learning: Explain that BLIP is a multimodal model trained on images paired with sentence descriptions. Describe, at a high level, how comparing image and caption embeddings and generating caption guesses helps it learn their relationship.
Why it matters: The paired-image training explains why BLIP can describe an image it has not seen before. - Caption generation pathway: Trace how BLIP generates a caption at inference time: an image encoder produces a descriptive numerical representation of the scene, and a text decoder turns that representation into a coherent sentence.
Why it matters: This two-part pathway connects the image supplied to BLIP with the sentence it returns.
Key terms: BLIP, multimodal model, embedding, image and caption pairs, inference, image encoder, text decoder, caption
2.4.1 BLIP in action
This section uses BLIP to caption familiar and generated images, then explains that its descriptions arise from learned patterns rather than handcoded rules.
- BLIP captions of generated images: Describe how BLIP captions the horse image and still identifies the zebra in a generated image, despite the zebra appearing in an unlikely setting.
Why it matters: The zebra example shows what the captioning model can do beyond a straightforward image. - Learned rather than hardcoded captions: Explain that the caption and its wording emerge from patterns in images and descriptions, not from horse- or zebra-specific rules or a handcoded grammar. State that the model involves multiple networks feeding into one another.
Why it matters: The example illustrates how learned patterns can produce a useful sentence without rules written for each subject.
Key terms: BLIP, captioning network, corpus of images and their descriptions, hardcoded criterion or grammar, multiple networks
2.5 Conclusion
The conclusion draws out why task-specific pretrained networks are useful, especially when adapting a model to a new task with limited data.
- Value of pretrained networks: Explain why a pretrained network can be a useful starting point for a new task, particularly when there are relatively few data points.
Why it matters: It explains why practitioners keep pretrained networks in their toolkit instead of always building a model from scratch.
Key terms: pretrained network, fine-tuning, data points
2.6 Exercises
2.7 Summary
Where this is used later
- The Vision Transformer (chapter 9) · Returns to the Vision Transformer in more depth.
- The PyTorch nn module (chapter 6) · Introduces PyTorch’s nn module in more depth.
- Named tensors (chapter 3) · The image tensor and added batch dimension used in the example.
- Dataset transforms (chapter 7) · Develops image dataset transforms in more depth.
- Measuring accuracy (chapter 8) · Evaluating a trained image classifier requires meaningful predictions.
- Output of a classifier (chapter 7) · Interpreting the output of another image classifier builds on the idea of class scores.
- Representing the output as probabilities (chapter 7) · The discussion of classifier probabilities develops the confidence-like interpretation introduced here.
- Training the classifier (chapter 7) · Training an image classifier gives context for why the examples it sees affect its predictions.
Concepts explained in this chapter
- foundational modelA neural network already trained on prior data and reused for inference or further training.
- inpaintingAn image-generation/editing technique that fills in masked regions of an input image using a model and conditioning prompt.
- maskMasking is the selective representation, concealment, inclusion, exclusion, or restriction of parts of data or a system according to specified criteria.
- promptUser-provided guidance such as points, boxes, or masks that steers the segmentation model toward a desired object or region.
- labelA target label restricted to one of two classes, used here in binary classification and metric computation.
- language modelA language model with enough scale to generate and answer with fluent text from prompts.
- pipelineA pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
- evaluation modeA model setting that switches layers such as dropout or batch normalization into inference behavior.
- parameterA learnable numeric value in a model.
- torch.nn.ModuleThe PyTorch base class for defining neural-network models as Python modules with parameters, submodules, and a forward method.
- transformsThe torchvision submodule containing image transformation utilities used for preprocessing and augmentation.
- diffusion modelGenerative models that create outputs by reversing a diffusion process, mentioned here as part of the Hugging Face model collection.
- embeddingA one-dimensional numeric data structure used here as a mathematical object on which sums and dot products can be computed.
- visual encoderThe part of a multimodal model that converts an image into a numeric representation for downstream decoding.
- denoisingThe process of removing noise from a corrupted input, here used iteratively in a diffusion model to reconstruct an image.
- Hugging FaceA repository service for discovering, sharing, and exchanging pretrained models and weights.
- Vision TransformerA transformer-based neural network adapted for computer vision tasks, especially image classification, by processing image patches with attention.
- image generationThe task of creating new visual content from learned patterns, mentioned here as a preceding example task.
- convolution kernelA small learned filter window applied over local regions of an input to compute weighted sums.
- preprocessingData preparation performed before model execution, here separating image and prompt encoding from the main model run.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.