liveBook · Manning

Semantic segmentation

As explained in Local Image Generation

A semantic-segmentation ControlNet is a conditioning model that guides image generation from a map assigning pixels to semantic classes.

Semantic segmentation gives image generation a compositional map instead of a rigid copy. It identifies what occupies each part of a scene—such as trees, sidewalks, and streets—so a ControlNet can preserve the arrangement while inventing the visual details. section 12.5 section 12.7

A practical semantic-segmentation workflow looks like this:

  • Start with a source image whose broad composition you want to retain. section 12.5
  • Run the image through the “OneFormer ADE20K Segmentor” preprocessing node. section 12.5.1
  • Let the preprocessor assign colored regions to recognized object categories. section 12.5.1
  • Feed that segmentation map into the semantic-segmentation ControlNet and generate with your chosen prompt. section 12.5.1

The useful trick is its level of abstraction. Scribble follows an outline closely; semantic segmentation records general regions and their meanings, not texture, appearance, or fine detail. The model therefore knows “trees go here” and “sidewalk goes there” while retaining room to create. section 12.5 section 12.5.1

Cherry-blossom street

Take a spring street photograph with cherry trees and a sidewalk. The segmentor maps the trees, street, and sidewalk into colored semantic regions. Generation can then preserve their relative placement while changing the scene’s style and details instead of reproducing the photograph exactly. section 12.5

How Local Image Generation covers it

How deep it goes
Explains preprocessing, 150 object labels, regional maps, and creative freedom, but does not detail model internals.
Examples
Uses a cherry-blossom street photograph, colored object regions, trees, streets, and sidewalks as concrete examples.
What it assumes
Assumes familiarity with source images and image generation, while introducing semantic labels and preprocessing through a concrete workflow.
What you can do after
Apply it via ComfyUI and the OneFormer ADE20K Segmentor to preserve scene composition while varying generated appearance.
How it connects
Connects semantic segmentation with ControlNet, ComfyUI, Stable Diffusion, preprocessing, and image composition.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Local Image Generation. The section links point to the parts of the book it draws on.