12 Using ControlNets
This chapter covers
- Expanding the power of ComfyUI with custom nodes installed through the Manager
- Creating hidden images with the QR Code controlnet
- Stylizing real photos to appear like anime using the Scribble controlnet
- Making user our human characters have just the right pose using Open Pose
- Copy the composition of a photo while allowing the model to be creative with Semantic Segmentation
“The grogginess was going away. He knew he hadn’t been drugged. He had let his imagination run away with him. He had been out of control”
-- Patricia Highsmith, The Talented Mr. Ripley
No matter how skilled we are at prompting, sometimes it can be frustrating getting Stable Diffusion, or any image generation model, to do exactly what we want. Despite trying a range of prompts and generating thousands of images, we often find we really don’t have control. In this chapter we’ll learn how to fix this using a variety of tools referred to as ControlNets. These are separate models, similar to the LoRAs covered in the last chapter, that will allow you a wide range of control over precisely how our images are generated. We’ll be able to hide images in our outputs, generate stylized versions of real photos, control the exact pose our characters are in and specify the general composition of objects in the scene.