Image generation computationally produces new images by learning visual latent spaces from image data, then sampling and decoding points from those spaces. It can create imaginary subjects, interpolate between real images, or respond to specified inputs. section 17.0 section 17.4
Use three principal model families for different generation and editing behaviors:
- Variational autoencoders learn structured, continuous latent representations suited to face swapping, expression changes, and smooth latent-space animation. section 17.4
- Diffusion models start with pure noise and repeatedly denoise it into realistic images; conditioning them on text enables text-to-image generation. section 17.2.5 section 17.4
- Generative adversarial networks provide a third major approach, alongside VAEs and diffusion models. section 17.4
Prompted generation with Stable Diffusion
A KerasHub TextToImage task can generate a 512×512 image from “A NASA astronaut riding an origami elephant in New York City.” A negative prompt such as “blue color” steers generation away from that attribute, while changing diffusion steps provides a practical quality-and-computation comparison. Outputs can still contain artifacts, such as duplicated tusks, because the model lacks complete anatomical and physical understanding. section 17.3
generate = function(num_images, diffusion_steps) {
noisy_images <- random_normal( #1
shape = c(num_images, self$image_size, self$image_size, 3),
seed = self$seed_generator
)
diffusion_times <- seq(1, 0, length.out = diffusion_steps)
for (i in seq_len(diffusion_steps - 1)) {
diffusion_time <- diffusion_times[i]
next_diffusion_time <- diffusion_times[i + 1]
.[noise_rates, signal_rates] <- diffusion_time |>
op_broadcast_to(c(num_images, 1, 1, 1)) |>
diffusion_schedule() #2
.[pred_images, pred_noises] <- #3
self$denoise(noisy_images, noise_rates, signal_rates) #3
.[next_noise_rates, next_signal_rates] <- next_diffusion_time |>
op_broadcast_to(c(num_images, 1, 1, 1)) |>
diffusion_schedule() #4
noisy_images <- #4
(next_signal_rates * pred_images) + #4
(next_noise_rates * pred_noises) #4
}
images <-
self$normalizer$mean + pred_images * op_sqrt(self$normalizer$variance) #5
op_clip(images, 0, 255)
}
)
Follow the generation loop from random noise through scheduled denoising steps to progressively higher-signal images. section 17.2.5
