A noise tensor is an array of noise values with a specified shape. In diffusion image generation, it is usually sampled from a standard normal distribution and used as the starting sample that denoising progressively transforms into an image. section 5.1.1 section 7.5
A noise tensor participates in diffusion generation as follows:
- Match the tensor shape to the clean image or latent representation being generated; latent Stable Diffusion samples use shape (4, 64, 64). section 5.1.1 section 10.4
- Sample random values, optionally using a seed so the initial tensor and resulting generation can be reproduced. section 7.4.1 section 10.4
- Feed the tensor into a trained denoising U-Net, then repeatedly apply a scheduler to reduce noise over inference steps. section 7.4 section 7.4.1
- Treat the initial tensor as a source of variation: different tensors produce different images, even with the same trained model. section 7.4.1 section 7.5
Interpolating two starting samples
Generate two images from different initial noise tensors, take weighted averages between those tensors, and denoise the intermediate tensors. The resulting images form a smooth visual transition between the originals. section 7.5 section 7.4
Listing 7.5 Defining a function to generate flower images
@torch.no_grad()
def generate(self,model,device,batch_size=1,generator=None,
eta=1.0,use_clipped_model_output=True,num_inference_steps=50):
imgs=[] #1
image=torch.randn((batch_size,model.in_channels,model.sample_size,
model.sample_size),
generator=generator).to(device) #2
self.set_timesteps(num_inference_steps)
for t in tqdm(self.timesteps): #3
model_output = model(image, t)["sample"]
image = self.step(model_output,t,image,eta,
use_clipped_model_output=use_clipped_model_output)
img = unnormalize_to_zero_to_one(image)
img = img.cpu().permute(0, 2, 3, 1).numpy()
imgs.append(img)
image = unnormalize_to_zero_to_one(image)
image = image.cpu().permute(0, 2, 3, 1).numpy()
return image, imgs #4
Inspect the generation function to see random tensor creation, device placement, scheduler timesteps, iterative denoising, and image conversion. section 7.4.1
