15 Fine-tuning Stable Diffusion 1.5
This chapter covers
- Using OneTrainer to setup training for fine-tuning
- Creating our own original checkpoint
- Learning how AI can be used to improve AI
- Understanding the details of setting up the full configuration for fine-tuning
- Analyzing our results to understand the benefits and limitations of our fine-tuning
Full fathom five thy father lies;
Of his bones are coral made;
Those are pearls that were his eyes:
Nothing of him that doth fade,
But doth suffer a sea-change
Into something rich and strange.
-- Shakespeare, The Tempest
We start this chapter with a famous passage from Shakespeare’s The Tempest. A fitting quote for our final chapter since The Tempest is often considered Shakespeare’s final play. The quote is also appropriate because in this chapter we will be making fundamental changes to our model by fine-tuning it to hopefully make it better, as “sea change” of our own sort.
Fine-tuning is the process that is used to create all original checkpoints (that aren’t created using the merging technique in the last chapter), where new data fundamentally changes the nature of the entire model. Very similar to training a LoRA, we will collect images and labels, use these to change the model and explore the results to find the ideal place to stop our fine-tuning. Unlike a LoRA we can expect everything about how our model behaves to be affected by the transformation.