liveBook · Manning

Embedding block

As explained in Build a Text-to-Image Generator (from Scratch)

An embedding block is a neural-network module containing two linear layers with a GELU activation that transforms time-step or label inputs into shaped embeddings.

An embedding block is a neural-network module that transforms a timestep or class-label input into a shaped embedding used by a denoising U-Net. It combines two linear layers with a GELU activation. section 6.2.1

The mechanism proceeds in three stages:

  • Reshape the result as spatial channels, such as (512, 1, 1) or (256, 1, 1), for decoder conditioning. section 6.2.1

Timestep and label embeddings

A timestep of 25 produces 512- and 256-channel embeddings after scaling by 1000. A three-item label batch produces corresponding 512- and 256-channel embeddings from one-hot class inputs.

How Build a Text-to-Image Generator (from Scratch) covers it

How deep it goes
Defines the linear-GELU-linear mechanism and traces tensor shapes, but does not detail learned parameter behavior.
Examples
Uses timestep 25 and labels 2, 9, and 0 to demonstrate 512- and 256-channel embedding outputs.
What it assumes
Assumes familiarity with neural-network layers, tensor shapes, timesteps, class labels, and one-hot vectors.
What you can do after
Build timestep and label embedding modules in PyTorch and reshape their outputs for U-Net conditioning.
How it connects
Connects diffusion models, denoising U-Nets, GELU activation, one-hot encoding, and conditional generation.
Cover of Build a Text-to-Image Generator (from Scratch)
liveBookmanning.com
Read Build a Text-to-Image Generator (from Scratch) on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Text-to-Image Generator (from Scratch). The section links point to the parts of the book it draws on.