A diffusion model generates an image by repeatedly removing small amounts of noise. Start with pure noise, run a denoising model in a loop, and a recognizable image can emerge. This reverses diffusion—the process of gradually adding noise until an image disappears. section 17.2
Generate an image with a pretrained model
In R, load Stable Diffusion 3 through KerasHub, then pass a prompt to `generate()`: ```r library(keras3) py_require("keras-hub==0.21.*") keras_hub <- import("keras_hub") task <- keras_hub$models$TextToImage$from_preset( "stable_diffusion_3_medium", image_shape = shape(512, 512, 3), dtype = "float16" ) image <- task$generate( "A NASA astronaut riding an origami elephant in New York City" ) plot(as.raster(image, max = max(image))) ``` The high-level `TextToImage` task handles tokenization and diffusion. Training a capable text-to-image model from scratch is expensive and time-consuming. section 17.3
Each denoising step has a *diffusion time*. In the described setup, time 1 is nearly all noise and time 0 is nearly all image signal. A *diffusion schedule* sets the noise and signal rates at each time; the example uses a cosine schedule for a smooth transition. section 17.2.3
Listing 17.7 The diffusion schedule
diffusion_schedule <- function(diffusion_times, min_signal_rate = 0.02,
max_signal_rate = 0.95) {
start_angle <- op_arccos(max_signal_rate) |> op_cast(dtype = "float32")
end_angle <- op_arccos(min_signal_rate) |> op_cast(dtype = "float32")
diffusion_angles <-
start_angle + diffusion_times * (end_angle - start_angle)
signal_rates <- op_cos(diffusion_angles)
noise_rates <- op_sin(diffusion_angles)
list(noise_rates = noise_rates, signal_rates = signal_rates)
}
The R function turns diffusion times into noise and signal rates. Its sine and cosine calculations keep the squared rates summing to 1. section 17.2.3
Two ways to use the same denoising idea are:
- Generate images without a prompt by sampling from noise. This is the unconditioned image-generation setup. section 17.1.1
- Condition generation on text: encode a prompt, provide its embedding alongside the noisy image, and let it guide denoising. Training uses prompt–image pairs; generation can use a new prompt. section 17.3
For a closer look at pretrained generation, separate it into prompt tokenization and encoding, progressive denoising from noise, and output scaling for display. Interpolating between two prompt embeddings before denoising can produce a smooth progression between their images. section 17.3.1
Diffusion models sit alongside variational autoencoders and generative adversarial networks as image-generation approaches. Their realistic output is useful, but realism is not understanding: generated images can still have extra fingers, inconsistent lighting, or incoherent objects. section 17.4 section 17.1.1
