11 Deeper Customization Using LoRA

 

This chapter covers

  • Limitations of creating checkpoints for diffusion models
  • Basics of what a LoRA is and how it works
  • Using LoRA to add styles to Flux
  • How Trigger Words can improve your LoRA result
  • Using a concept LoRA to create a consistent character
  • Combining multiple LoRAs together
“Before the law sits a gatekeeper. To this gatekeeper comes a man from the country who asks to gain entry into the law. But the gatekeeper says that he cannot grant him entry at the moment. The man thinks about it and then asks if he will be allowed to come in later on. ‘It is possible,’ says the gatekeeper, ‘but not now.’”

-- Franz Kafka, Before the Law

In this chapter we’re going to learn about a very powerful way to get Flux and Stable Diffusion to create images with a much wider variety of styles as well as other customizations like creating a character or even being able to generate images of yourself! The technique, called LoRA (Low Rank Adaptation), uses a much smaller model to create dramatic changes to the output of a diffusion model. This is especially important for Flux which lacks the variety of fine-tuned checkpoints we explored in Chapter X with Stable Diffusion. Before diving into it, let’s chat a bit more about the challenges of working with diffusion models that make LoRAs a necessity.

11.1 VRAM and Diffusion Models

11.2 What are LoRAs?

11.3 Visualizing Kafka

11.4 Using Your First LoRA

11.4.1 Setting up the LoRA workflow

11.4.2 Modifying LoRA Strength

11.4.3 A Tarot Card LoRA

11.4.4 Using Trigger Words

11.5 Concept LoRAs

11.6 Combining LoRAs

11.7 Conclusion

11.8 Summary