liveBook · Manning

Fine-tuning

As explained in Local Image Generation

Fine-tuning is the continued training of a pretrained model on additional data or a specialized task to adapt its behavior.

Fine-tuning is extra training for a pretrained diffusion model: give a base model a new image collection and it learns to generate that kind of material better. It is far cheaper than training from scratch, which is fortunate because full training can demand absurd amounts of compute. Fine-tuning changes the model itself—here, the U-NET—so the result is an original custom checkpoint rather than a small add-on. section 5.1.1 section 15.0

A checkpoint is a saved stopping point in training, and choosing one is part of the job. More epochs do not automatically mean better images: inspect samples from several points, then keep the version whose results are strongest and most consistent for the prompts you care about. section 5.1.1 section 15.5.1 section 15.6

A practical fine-tuning loop is:

  • Choose a base model and a goal that exposes a real weakness, such as better sci-fi and horror monsters. section 15.1
  • Collect training images and labels; the image set may be generated with AI rather than assembled manually. section 15.0
  • Configure a training task in OneTrainer, selecting SD1.5 and Fine Tune, then choose a roomy workspace for checkpoints, samples, backups, and configuration files. section 15.3.2
  • Run training, save intermediate checkpoints, and compare generated samples across epochs instead of trusting a dashboard alone. section 15.3.2 section 15.5
  • Refine with more data from a promising checkpoint when the first run improves results but still leaves room to grow. section 15.6

Teaching SD 1.5 better movie monsters

An unmodified SD 1.5 prompt for a monstrous grasshopper attacking a city produces a loose, incoherent result. After fine-tuning, the model produces a giant grasshopper-like monster towering over a city, much closer to the request. The gains are not uniform: a crab-monster prompt was already good before training, while some sandworm results improved slowly and still missed prompt adherence. section 15.1 section 15.5.1

Fine-tuning can also affect behavior outside the narrow-looking training goal. Human-focused prompts improved quickly: a malformed space man became recognizable after 10 epochs, and a cowboy’s face cleaned up substantially within 20–30 epochs. That broad change is powerful, but it means every checkpoint deserves testing on varied prompts. section 15.5.2 section 15.0

Figure 15.2 The result of the fine-tuning we’ll be doing this chapter looks much closer to our prompt

Compare the fine-tuned grasshopper result with the original to see the visible target of training: clearer subject, scale, and city scene. section 15.1

Hardware sets the practical boundary. SD 1.5 can be fine-tuned on an RTX 4090, while Flux fine-tuning is currently much more demanding and expensive. If the aim is to experiment with custom checkpoints at home, SD 1.5 is the friendlier starting point; LoRAs are a lighter alternative because they modify only attention layers and can be trained on consumer hardware. section 9.2.1 section 9.3 section 11.8

How Local Image Generation covers it

How deep it goes
Walks through data, OneTrainer setup, checkpoints, epoch-by-epoch samples, and result analysis without detailing training internals.
Examples
Compares monster, space-man, and cowboy prompts across training epochs, including grasshopper, sandworm, and crab outputs.
What it assumes
Teaches the training workflow from practical settings while assuming familiarity with Stable Diffusion image generation and prompts.
What you can do after
Operate/configure it with OneTrainer to create and evaluate an SD1.5 custom checkpoint from image-and-label training data.
How it connects
Connects fine-tuning with Stable Diffusion, U-NETs, checkpoints, LoRAs, ComfyUI, Tensorboard, and quantization.

Also explained in

Compare how 52 books explain fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Local Image Generation. The section links point to the parts of the book it draws on.