Cover of Build a Text-to-Image Generator (from Scratch)

What you will learn in Build a Text-to-Image Generator (from Scratch)

With transformers and diffusions

by Mark Liu

Read it on liveBook Get the book at manning.com

Chapter by chapter

Chapter 1: A tale of two models: Transformers and diffusions

1.1 What is a text-to-image generation model? · 1.1.1 Unimodal vs. multimodal models · 1.1.2 Practical use cases of text-to-image models · 1.2 Transformer-based text-to-image generation · 1.2.1 Converting an image into a sequence of integers and then back · 1.2.2 Training and using a transformer-based text-to-image model · 1.3 Text-to-image generation with diffusion models · 1.3.1 Forward and reverse diffusions · 1.3.2 Latent diffusion models and Stable Diffusion · 1.4 How to build text-to-image models from scratch · 1.5 Challenges for text-to-image generation models · 1.5.1 Are generative AI models stealing from artists? · 1.5.2 The geometric inconsistency problem · 1.6 Social, environmental, and ethical concerns · 1.7 Summary

What you learn in chapter 1 →

Chapter 2: Build a transformer

2.1 An overview of attention and transformers · 2.1.1 How the attention mechanism works · 2.1.2 How to create a transformer · 2.2 Word embedding and positional encoding · 2.2.1 Word tokenization with the Spacy library · 2.2.2 A sequence padding function · 2.2.3 Input embedding from word embedding and positional encoding · 2.3 Creating an encoder–decoder transformer · 2.3.1 Coding the attention mechanism · 2.3.2 Defining the Transformer() class · 2.3.3 Creating a language translator · 2.4 Training and using the German-to-English translator · 2.4.1 Training the encoder–decoder transformer · 2.4.2 Translating German to English with the trained model · 2.5 Summary

What you learn in chapter 2 →

Chapter 3: Classify images with a vision transformer

3.1 The blueprint to train a ViT · 3.1.1 Converting images to sequences · 3.1.2 Training a ViT for classification · 3.2 The CIFAR-10 dataset · 3.2.1 Downloading and visualizing CIFAR-10 images · 3.2.2 Preparing datasets for training and testing · 3.3 Building a ViT from scratch · 3.3.1 Dividing images into patches · 3.3.2 Modeling the positions of different patches in an image · 3.3.3 Using the multi-head self-attention mechanism · 3.3.4 Building an encoder-only transformer · 3.3.5 Using the ViT to create a classifier · 3.4 Training and using the ViT to classify images · 3.4.1 Choosing the optimizer and the loss function · 3.4.2 Training the ViT for image classification · 3.4.3 Classifying images using the trained ViT · 3.5 Summary

What you learn in chapter 3 →

Chapter 4: Add captions to images

4.1 Training and using a transformer to add captions · 4.1.1 Preparing data and the causal attention mask · 4.1.2 Creating and training a transformer · 4.2 Preparing the training dataset · 4.2.1 Downloading and visualizing Flickr 8k images · 4.2.2 Building a vocabulary of tokens · 4.2.3 Preparing the training dataset · 4.3 Creating a multimodal transformer to add captions · 4.3.1 Defining a ViT as the image encoder · 4.3.2 Creating the decoder to generate text · 4.4 Training and using the image-to-text transformer · 4.4.1 Training the encoder–decoder transformer · 4.4.2 Adding captions to images with the trained model · 4.5 Summary

What you learn in chapter 4 →

Chapter 5: Generate images with diffusion models

5.1 The forward diffusion process · 5.1.1 How diffusion models work · 5.1.2 Visualizing the forward diffusion process · 5.1.3 Different diffusion schedules · 5.2 The reverse diffusion process · 5.3 A blueprint to train the U-Net model · 5.3.1 Steps in training a denoising U-Net model · 5.3.2 Preprocessing the training data · 5.4 Training and using the diffusion model · 5.4.1 The Denoising Diffusion Probabilistic Model noise scheduler · 5.4.2 Inference using the U-Net denoising model · 5.4.3 Training and using the denoising U-Net model · 5.5 Summary

What you learn in chapter 5 →

Chapter 6: Control what images to generate in diffusion models

6.1 Classifier-free guidance in diffusion models · 6.1.1 An overview of classifier-free guidance · 6.1.2 A blueprint to implement CFG · 6.2 Different components of a denoising U-Net model · 6.2.1 Time step embedding and label embedding · 6.2.2 The U-Net denoising model architecture · 6.2.3 Down blocks and up blocks in the U-Net · 6.3 Building and training the denoising U-Net model · 6.3.1 Building the denoising U-Net · 6.3.2 The Denoising Diffusion Probabilistic Model · 6.3.3 Training the diffusion model · 6.4 Generating images with the trained diffusion model · 6.4.1 Visualizing generated images · 6.4.2 How the guidance parameter affects generated images · 6.5 Summary

What you learn in chapter 6 →

Chapter 7: Generate high-resolution images with diffusion models

7.1 Attention in U-Net, DDIM, and image interpolation · 7.1.1 Incorporating the attention mechanism in the U-Net model · 7.1.2 Denoising Diffusion Implicit Models · 7.1.3 Image interpolation in diffusion models · 7.2 High-resolution flower images as training data · 7.2.1 Visualizing images in the training dataset · 7.2.2 Applying forward diffusion on flower images · 7.3 Building and training a U-Net for high-resolution images · 7.3.1 Building the denoising U-Net model · 7.3.2 Training the denoising U-Net model · 7.4 Image generation and interpolation · 7.4.1 Using the trained denoising U-Net to generate images · 7.4.2 Transition from one image to another · 7.5 Summary

What you learn in chapter 7 →

Chapter 8: CLIP: A model to measure the similarity between image and text

8.1 The CLIP model · 8.1.1 How the CLIP model works · 8.1.2 Selecting an image from Flickr 8k based on a text description · 8.2 Preparing the training dataset · 8.2.1 Image-caption pairs in Flickr 8k · 8.2.2 The DistilBERT tokenizer · 8.2.3 Preprocess captions and images for training · 8.3 Creating a CLIP model · 8.3.1 Creating a text encoder · 8.3.2 Creating an image encoder · 8.3.3 Building a CLIP model · 8.4 Training and using the CLIP model · 8.4.1 Training the CLIP model · 8.4.2 Using the trained CLIP model to select images · 8.4.3 Using the OpenAI pretrained CLIP model to select images · 8.5 Summary

What you learn in chapter 8 →

Chapter 9: Text-to-image generation with latent diffusion

9.1 What is a latent diffusion model? · 9.1.1 How variational autoencoders work · 9.1.2 Combining a latent diffusion model with a variational autoencoder · 9.2 Compressing and reconstructing images with VAEs · 9.2.1 Downloading the pretrained VAE · 9.2.2 Encoding and decoding images with the pretrained VAE · 9.3 Text-to-image generation with latent diffusion · 9.3.1 Guidance by the CLIP model · 9.3.2 Diffusion in the latent space · 9.3.3 Converting latent images to high-resolution ones · 9.4 Modifying existing images with text prompts · 9.5 Summary

What you learn in chapter 9 →

Chapter 10: A deep dive into Stable Diffusion

10.1 Generating images with Stable Diffusion · 10.2 The Stable Diffusion architecture · 10.2.1 Generating images from text with Stable Diffusion · 10.2.2 Text embedding interpolation · 10.3 Creating text embeddings · 10.4 Image generation in the latent space · 10.5 Converting latent images to high-resolution ones · 10.6 Summary

What you learn in chapter 10 →

Chapter 11: VQGAN: Convert images into sequences of integers

11.1 Converting images into sequences of integers and back · 11.2 Variational autoencoders · 11.2.1 What is an autoencoder? · 11.2.2 The need for VAEs and their training methodology · 11.3 Vector quantized variational autoencoders · 11.3.1 The need for VQ-VAEs · 11.3.2 The VQ-VAE model architecture and training process · 11.4 Vector quantized generative adversarial networks · 11.4.1 Generative adversarial networks · 11.4.2 VQGAN: A GAN with a VQ-VAE generator · 11.5 A pretrained VQGAN model · 11.5.1 Reconstructing images with the pretrained VQGAN · 11.5.2 Converting images into sequences of integers · 11.6 Summary

What you learn in chapter 11 →

Chapter 12: A minimal implementation of DALL-E

12.1 How min-DALL-E works · 12.1.1 Training min-DALL-E · 12.1.2 From prompt to pixels: Image generation at inference time · 12.2 Tokenizing and encoding the text prompt · 12.2.1 Tokenizing the text prompt · 12.2.2 Encoding the text prompt · 12.3 Iterative prediction of image tokens · 12.3.1 Loading the pretrained BART decoder · 12.3.2 Predicting image tokens using the BART decoder · 12.4 Converting image tokens to high-resolution images · 12.4.1 Loading the pretrained VQGAN detokenizer · 12.4.2 Visualizing the intermediate and final high-resolution outputs · 12.5 Summary

What you learn in chapter 12 →

Chapter 13: New developments and challenges in text-to-image generation

13.1 State-of-the-art text-to-image generators · 13.1.1 DALL-E series · 13.1.2 Google’s Imagen · 13.1.3 Latent diffusion models: Stable Diffusion and Midjourney · 13.2 Challenges and concerns · 13.3 A blueprint to fine-tune ResNet50 · 13.3.1 The history and architecture of ResNet50 · 13.3.2 A plan to fine-tune ResNet50 for classification · 13.3.3 Using ResNet50 to classify images · 13.4 Fine-tuning ResNet50 to detect fake images · 13.4.1 Downloading and preprocessing real and fake face images · 13.4.2 Fine-tuning ResNet50 · 13.4.3 Detecting deepfakes using the fine-tuned ResNet50 · 13.5 Summary

What you learn in chapter 13 →

Concepts explained in this book

embeddingAn embedding is a numerical vector representation of an input or discrete item in a learned feature space.
decodeA sequence decoder is a neural-network component that generates an output sequence from contextual representations and previously generated elements.
encoderA sequence encoder is a neural-network component that maps an input sequence to contextualized representations for subsequent processing.
latent spaceA latent space is a lower-dimensional representation space in which encoded data is represented before decoding or further generation.
U-NetU-Net is an encoder-decoder neural-network architecture commonly used as the denoising backbone in diffusion models.
diffusion modelA denoising diffusion probabilistic model is a diffusion generative model that samples through a probabilistic sequence of noise-removal steps.
variational autoencoderA variational autoencoder is a generative neural model that encodes data into a lower-dimensional probabilistic latent representation and decodes that representation into reconstructed or generated data.
transformer architectureA transformer architecture is a neural-network architecture that uses attention mechanisms to process relationships among elements in a sequence or other structured input.
text-to-image modelA text-to-image model is a machine-learning model that generates images from textual descriptions.
Vision TransformerA Transformer model that processes an image as a sequence of patch representations for computer-vision tasks.
VQGANA vector-quantized generative adversarial network that combines discrete latent representations with adversarial and perceptual image modeling.
positional embeddingA positional encoding is a representation added to sequence-element embeddings to convey their positions or ordering.
attentionA neural-network mechanism that weights input elements according to their relevance when computing a representation or prediction.
CLIPContrastive language-image pretraining is a training technique that aligns image and text representations by contrasting matching and nonmatching pairs.
promptA text prompt is a natural-language instruction or description that specifies desired generated content.
multi-head attentionAn attention mechanism that computes attention through multiple separate query, key, and value subspaces to capture different relationships simultaneously.
denoisingA denoising process is a procedure that progressively removes noise from a data sample to produce a clean result.
patchAn image patch is a spatially localized portion of an image represented separately in a grid or latent representation.
codebookA codebook is a learned finite collection of discrete vectors used as representatives for continuous latent vectors.
image tokenizationImage tokenization is the process of converting image content into a sequence of discrete visual tokens.
autoregressive modelA process that produces an ordered output sequence one element at a time, conditioning each element on previously generated elements and available conditioning information.
encoder-decoder transformerAn encoder-decoder transformer is a transformer model with an encoder that represents an input sequence and a decoder that generates an output sequence from that representation.
residual connectionA direct link that transfers representations between corresponding layers or stages of a neural network.
text encoderA text encoder is a neural-network component that maps natural-language text to a numerical representation for downstream conditioning or processing.
latent diffusion modelA latent diffusion model is a diffusion model that performs forward and reverse diffusion in a compressed latent representation rather than directly in pixel space.
VQGAN decoderA VQGAN decoder is the image-reconstruction component that maps discrete image tokens to an output image.
CFG scaleCFG scale is a sampling parameter that controls the strength with which classifier-free guidance follows conditioning information.
sequenceSequential data is data whose elements have an order that carries information about their meaning or interpretation.
[CLS] tokenA class token, or CLS token, is a learned sequence element whose contextual representation supports sequence-level or image-level classification.
image-text pairAn image-text pair is a paired example consisting of an image and an associated textual description or caption.
scaled-dot product attentionScaled dot-product attention computes attention weights from scaled dot products between query and key vectors.
tokenA token is a discrete unit produced by segmenting text, such as a word, punctuation mark, or subword.
noise scheduleA sampling configuration component that determines how noise levels and transitions are applied across diffusion inference steps.
ResNet50ResNet50 is a 50-layer residual convolutional neural network commonly used for image recognition and visual feature extraction.
interpolationInterpolation is the construction of intermediate values or representations between specified endpoints by varying their contributions.
masked attentionMulti-head self-attention that restricts each position from accessing subsequent positions in the sequence.
classifier-free guidanceClassifier-free guidance is a diffusion-sampling technique that steers generation by combining conditional and unconditional predictions from the diffusion model itself rather than using an external classifier.
cross-attentionAn attention mechanism that computes attention between decoder representations and representations from an encoded conditioning input.
vector-wise quantizationVector quantization maps a continuous vector to a discrete representative selected from a codebook.
beginning-of-sequence tokenA beginning-of-sentence token is a special sequence marker that indicates where a generated sentence starts.
end of text tokenAn end-of-sequence token is a special marker indicating the end of a token sequence.
latent imageA latent image is an image-like tensor represented in the compressed latent space of an image-generation model rather than directly in pixel space.
next-token predictionNext-token prediction is a sequence-generation technique that predicts the next discrete token from preceding tokens and conditioning information.
vector quantized variational autoencoderA vector quantized variational autoencoder is a variational autoencoder that represents inputs with discrete learned codebook vectors.
training datasetA training subset is the portion of a dataset supplied to a model for learning its parameters or representations.
Stable DiffusionStable Diffusion is a latent diffusion model for generating and editing images from text and other conditioning inputs.
noiseNoise is random variation added to an image that obscures its original visual information.
noise tensorA noise tensor is a tensor containing noise values with a specified shape, here serving as the starting sample for image generation.
fine-tuningFine-tuning is adapting a pretrained model to a new task by updating selected parameters, often only the task-specific output layers.
noisy imageA noisy image is an image formed by combining an underlying image with injected noise.
resolutionImage resolution is the spatial dimensions and corresponding level of discrete visual detail represented by an image.
min-DALL-EA compact Transformer-based text-to-image model that predicts discrete image tokens autoregressively from text.
VQGAN encoderA VQGAN encoder is the image-processing component that maps an image to a sequence of discrete image tokens.
transposed convolutionA transposed convolutional layer increases spatial resolution by applying a learned convolution-like operation that can expand spatial dimensions.
discrete codeA discrete latent vector is a latent representation whose components are selected from a finite set of learned codebook vectors.
feedforward networkA feed-forward network that applies the same transformation independently to each position in a sequence.
shared representationA joint embedding space is a shared representation space in which different data modalities are encoded so that corresponding items have related representations.
BART encoderA BART encoder is the component that transforms a text caption or prompt into an encoded representation for sequence generation.
sequence predictionA prediction task in which a model generates or estimates elements of an ordered sequence.
Transformer encoder layerAn encoder layer is a transformer block that combines self-attention and a position-wise feed-forward network, typically with normalization and residual connections.
pretrained modelPretrained weights are model parameter values learned previously on another task or dataset and reused for later adaptation.
CIFAR-10CIFAR-10 is a labeled image dataset containing color images divided into 10 classes for machine-learning evaluation and training.
tokenizerA tokenizer is a text-processing component that segments a character string into discrete tokens or token identifiers.
keyA key is an attention vector representing information against which a query is compared to determine relevance.
queryA query is an attention vector representing the information whose relevance to sequence elements is being evaluated.
valueA value is an attention vector whose information is combined according to attention weights.
reconstruction lossA reconstruction loss measures the discrepancy between input data and the data reconstructed from its latent representation.
batchingA batch is a group of training examples processed together in one model update or evaluation step.
padding tokenA padding token is a special marker appended to shorter sequences so that sequences in a batch have equal length.
max poolingMax pooling is a spatial downsampling operation that retains the maximum activation within each local window.
Flickr 8k datasetThe Flickr 8k Dataset is a collection of 8,000 photographs, each paired with five human-written captions for image-captioning research.
clean imageA clean image is an image containing the original visual signal without diffusion noise.
encoder-only transformerAn encoder-only transformer consists of encoder layers that convert an input sequence into continuous vector representations.
layer normalizationLayer normalization normalizes the activations within each example to stabilize neural-network computation and training.
softmaxA function that converts a vector of real-valued scores into nonnegative values that sum to one.
tokenizationTokenization is the process of splitting text into discrete tokens such as words, punctuation marks, or subwords.
quantization lossA quantization loss measures the discrepancy between continuous latent vectors and their selected discrete codebook representatives.
attention_maskA tensor or pattern that hides selected sequence positions from a model during processing or prediction.
BARTBART is a denoising sequence-to-sequence transformer model whose encoder is bidirectional and whose decoder is autoregressive.
DALL-EA text-to-image generative model that produces images conditioned on natural-language descriptions.
evaluation datasetA validation set is a held-out dataset split used to assess model performance during training and select configurations or checkpoints.
deep fakeA deepfake is synthetic or manipulated media generated or altered with machine-learning methods to impersonate or misrepresent real people or objects.
DDIMA DDIM scheduler is a configuration component that determines the noise levels and time-step transitions for sampling with DDIM.
tensorA tensor is a multidimensional numerical array used to represent data and intermediate computations in a neural network.
label embeddingA label embedding is a learned vector or tensor representation of a class label supplied to a conditional denoising network.
down blockA down block is a U-Net component that extracts features while reducing spatial resolution through convolutional and pooling operations.
up blockAn up block is a U-Net component that increases spatial resolution through operations such as transposed convolution.
bottleneck layerA bottleneck layer is the narrow intermediate representation connecting the contracting and expansive paths of an encoder-decoder network.
deepfake detectionDeepfake detection is the analysis of digital media to determine whether it has been synthetically generated or manipulated using machine-learning methods.
objective functionA loss function is a function that quantifies the discrepancy between a model's predictions and the desired outputs.
image captioningImage captioning is the task of generating natural-language descriptions of image content.
timestepA time step is a discrete stage in the diffusion process at which a particular noise level is applied.
unconditional promptAn unconditional prompt is an empty text input representing the absence of a specific conditioning description.
feature vectorA feature vector is a numerical vector whose components encode properties of an input or intermediate representation.
linear noise scheduleA linear diffusion schedule is a noise schedule in which the noise parameter changes linearly across diffusion time steps.
reconstructed imageA reconstructed image is an image recovered by decoding a compressed or latent representation.
diffusersDiffusers is an open-source software library providing implementations and pipelines for diffusion models and their image-generation workflows.
maskA mechanism that selectively blocks information from specified sequence positions during model computation.
poolingA pooling layer aggregates values within local spatial windows to produce a reduced or transformed feature representation.
timestep embeddingA time embedding is a learned vector or tensor representation of a diffusion time step supplied to a denoising network.
token idA token ID is an integer identifier assigned to a token in a vocabulary.
DALL-E miniDALL-E mini is an open-source project that replicates the text-to-image behavior of DALL-E.
image-to-text generationImage-to-text generation is the creation of natural-language descriptions from visual input.
cross entropy lossCategorical cross-entropy loss measures the discrepancy between predicted class probabilities and the ground-truth class in multicategory classification.
accuracyAccuracy is the proportion of predictions that exactly match the corresponding ground-truth labels.
contrastive learningContrastive learning is a representation-learning technique that brings related examples closer and separates nonmatching examples in an embedding space.
contracting encoder pathA contracting path is the encoder-side portion of a U-Net that progressively reduces spatial resolution while learning increasingly abstract feature representations.
expansive decoder pathAn expansive path is the decoder-side portion of a U-Net that increases spatial resolution and reconstructs an image from condensed features.
attention weightA normalized attention score that determines how strongly a value contributes to an attention output.
denoising modelA denoising model estimates or removes noise from a corrupted data sample.
image selectionImage selection is the task of choosing the image that best matches a textual description from a collection of images.
multimodalA generative model that processes or connects two or more data modalities, such as text, images, audio, or video.
image-to-text transformerA transformer model that encodes an image and generates a natural-language description of it.
AdamWAdamW is an adaptive gradient-based optimizer that decouples weight decay from the parameter update.
decoder-only transformerA decoder-only transformer consists of decoder layers that autoregressively generate output sequences from preceding tokens.
logitA logit is an unnormalized real-valued score assigned to a class before conversion into probabilities.
downsamplingDownsampling is the reduction of the spatial resolution of a representation.
geometric consistencyThe preservation of coherent spatial structure, proportions, and object geometry across objects, scenes, and viewpoints.
geometric inconsistency problemThe geometric inconsistency problem is the difficulty of producing generated images whose spatial structure and geometric relationships are coherent.
parameterA model parameter is a numerical value used by a model that is typically learned or adjusted during training.
discriminatorA discriminator is a neural-network component that distinguishes generated samples from samples drawn from the target data distribution.
perceptual lossA perceptual loss measures differences between images using features intended to reflect human visual similarity rather than only pixel-wise differences.
biasModel bias is a systematic tendency in a model's outputs that reflects or amplifies unequal or stereotypical patterns associated with social groups or cultural contexts.
environmental impactThe effects that an activity or technology has on the natural environment, including through resource and energy use.
misinformationFalse or inaccurate information that is presented or circulated as factual, regardless of whether deception is intended.
language modelA language model is a model that processes or generates human language, typically as sequences of text.
generative modelingGenerative AI is a class of machine-learning models that creates new content resembling patterns learned from data.
convolutional layerConv2d is a two-dimensional convolution layer that applies learned spatial filters to image-like tensors.
adversarial trainingAdversarial training is a training procedure in which competing models are optimized against each other, typically with a generator producing samples and a discriminator judging them.
linear layerA fully connected layer is a neural-network layer in which every output unit receives input from every unit in the preceding layer.
generatorA GAN generator is a neural network that transforms random input noise into synthetic samples intended to resemble real data.
VQ-VAE decoderA VQ-VAE decoder is the image-reconstruction component that maps quantized latent codes or their codebook vectors back to an image.
top-k samplingTop-k sampling draws the next token from a probability distribution restricted to the k highest-probability candidates.
out-of-vocabulary wordAn unknown token is a special marker representing a token not present in the model's vocabulary.
attention matrixAn attention map is a matrix of attention probabilities showing how strongly tokens attend to one another.
visual encoderAn image encoder is a neural-network component that maps an image to a numerical representation.
quantized latent representationA quantized latent representation is a latent representation whose continuous vectors have been replaced by discrete codebook vectors.
early stoppingEarly stopping is a training technique that halts optimization when a model fails to improve for a specified number of epochs.
single-modal modelA unimodal model processes data within a single modality, such as text-only or image-only data.
DistilBERTDistilBERT is a smaller and faster transformer-based language model distilled from BERT while retaining much of its language understanding capability.
mean absolute errorMean absolute error, also called L1 loss, measures the average absolute difference between predicted numerical values and their target values.
attention scoreAn attention score is a numerical measure of the relevance between a query and a key.
geometryThe three-dimensional organization and shape of objects and their spatial arrangement in a scene.
spatial informationThe relative positional or geometric arrangement between objects or parts of a scene.
intellectual propertyLegal rights protecting creations of the mind, including artistic works and inventions.
VisionEncoderDecoderVisionEncoderDecoder is a local model class that combines an image encoder with a text decoder for image-conditioned text generation.
classifierClassification is the task of assigning an input to one or more predefined categories.
transformer-based text-to-image modelA transformer-based text-to-image model generates images from text by modeling relationships between textual and visual token sequences with transformer architectures.
StableDiffusionPipelineStableDiffusionPipeline is a Diffusers pipeline component that runs Stable Diffusion text-to-image generation from a prompt.
cosine noise scheduleA cosine diffusion schedule is a noise schedule whose clean-image and noise weights vary over time according to a cosine-based function.
image generationImage synthesis is the computational generation of new images from learned representations or specified conditions.
conditioningConditioning information is auxiliary input supplied to a generative process to influence the characteristics of its output.
probability distributionA probability distribution describes the possible values of a random variable and the probabilities associated with those values.
normal distributionA normal distribution is a continuous probability distribution characterized by a bell-shaped density determined by its mean and variance.
label maskingLabel masking is the probabilistic removal of class-label conditioning during training so a model learns both conditional and unconditional predictions.
embedding blockAn embedding block is a neural-network module containing two linear layers with a GELU activation that transforms time-step or label inputs into shaped embeddings.
categorical encodingA one-hot tensor represents each categorical value with a vector containing one active entry and zeros elsewhere.
transformer blockA transformer block is a neural-network component that combines attention sublayers with a feed-forward network for sequence processing.
PyTorch UnfoldPyTorch Unfold is a tensor operation that extracts sliding local blocks from an input tensor.
Fashion MNISTFashion MNIST is a dataset of 70,000 grayscale images of clothing items such as shirts, shoes, and bags.
classifier guidanceClassifier guidance is a diffusion-sampling technique that uses a separately trained classifier to steer intermediate samples toward a desired class or condition.
computational footprintThe total computational resource requirement associated with operating a model or system.
energy consumptionThe amount of electrical energy consumed by computing processes such as model training and inference.
VQ-VAE generatorA VQ-VAE generator is the VQGAN component consisting of an encoder, a codebook, and a decoder that reconstructs images from discrete latent codes.
LPIPSLPIPS is a learned perceptual image similarity metric that compares images using features extracted by pretrained neural networks.
mean squared errorMean squared error is the average of the squared differences between predicted and reference numerical values.
adversarial lossAn adversarial loss is a training objective that evaluates how convincingly generated samples resemble real samples in a generative adversarial setup.
generative adversarial networkA generative adversarial network is a generative model trained through competition between a generator that creates samples and a discriminator that distinguishes generated samples from real data.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.