A timestep embedding turns a diffusion timestep into a numerical vector that tells the model how much noise an image contains. Slightly noisy images need gentle denoising, while nearly static images need stronger denoising. section 12.10
The embedding uses sinusoidal encoding, similar to positional encoding. Sine and cosine values calculated at different scales give each timestep a distinctive vector while representing its position along the noise schedule. section 12.10
A simple construction follows these steps:
- Choose a timestep, such as t = 22, and an embedding size, such as 16. section 12.10
- Calculate a scaled angle for each vector position using increasing divisors. section 12.10
- Apply alternating sine and cosine functions, then store the results as the embedding vector. section 12.10
- Pass the generic vector through a feed-forward network to produce a representation tailored to the diffusion model. section 12.11
Timestep 22
For t = 22, the calculation produces eight sine values and eight cosine values, concatenated into a 16-dimensional vector. During denoising, this vector is combined with the text embedding and added to the U-Net bridge tensor. Each embedding value is broadcast across one feature-map channel, allowing the network to condition its prediction on the current noise level. section 12.10 section 12.18
Table 12.2 Calculations for creating timestep embeddings
|
Index (i) |
angle calculation |
angle |
sin(angle) |
cos(angle) |
| 0 |
22 / 1 |
22 |
-0.0089 |
-0.9999 |
| 1 |
22 / 3.16 |
6.957 |
-0.6277 |
0.7784 |
| 2 |
22 / 10 |
2.2 |
0.8085 |
-0.5885 |
| 3 |
22 / 31.62 |
0.696 |
0.6411 |
0.7674 |
| 4 |
22 / 100 |
0.22 |
0.2182 |
0.9759 |
| 5 |
22 / 316.23 |
0.070 |
0.0699 |
0.9975 |
| 6 |
22 / 1000 |
0.022 |
0.0220 |
0.9997 |
| 7 |
22 / 3162.23 |
0.007 |
0.0070 |
0.9999 |
Use the calculation table to trace timestep 22 through scaled angles, sine values, and cosine values. section 12.10
At inference, a new timestep embedding is generated at every denoising step. The U-Net uses it with the text embedding to predict noise, gradually refining an initially random image. section 12.25
