11 Deeper Customization Using LoRA
This chapter covers
- Limitations of creating checkpoints for diffusion models
- Basics of what a LoRA is and how it works
- Using LoRA to add styles to Flux
- How Trigger Words can improve your LoRA result
- Using a concept LoRA to create a consistent character
- Combining multiple LoRAs together
“Before the law sits a gatekeeper. To this gatekeeper comes a man from the country who asks to gain entry into the law. But the gatekeeper says that he cannot grant him entry at the moment. The man thinks about it and then asks if he will be allowed to come in later on. ‘It is possible,’ says the gatekeeper, ‘but not now.’”
-- Franz Kafka, Before the Law
In this chapter we’re going to learn about a very powerful way to get Flux and Stable Diffusion to create images with a much wider variety of styles as well as other customizations like creating a character or even being able to generate images of yourself! The technique, called LoRA (Low Rank Adaptation), uses a much smaller model to create dramatic changes to the output of a diffusion model. This is especially important for Flux which lacks the variety of fine-tuned checkpoints we explored in Chapter X with Stable Diffusion. Before diving into it, let’s chat a bit more about the challenges of working with diffusion models that make LoRAs a necessity.