Encoder

As explained in Learn Generative AI with PyTorch

An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.

An encoder turns an input into a learned representation that another component can use. In an autoencoder, it compresses an image into a latent vector for a decoder to reconstruct. In an encoder–decoder Transformer, it turns an input sentence into vectors that help a decoder produce a translation. The representation is the encoder’s output; reconstruction or generation is the decoder’s job. section 7.1.1 section 1.3.2

Compress handwritten digits

Start with a 28 × 28 grayscale digit. Flatten its 784 pixels and pass them through two dense layers to obtain a 20-value encoding. A decoder maps that vector back to 784 pixel values. To train both parts, compare the reconstruction with the original image, backpropagate the pixel-level loss, and inspect the reconstructed digits. The 20-value size is a design choice, not a special property of digits. section 7.1.2 section 7.2.2

Figure 7.3 The architecture of an AE and the steps to train one to generate handwritten digits. An AE consists of an encoder (middle left) and a decoder (middle right). In each iteration of training, images of handwritten digits are fed to the encoder (step 1). The encoder compresses the images to deterministic points in the latent space (step 2). The decoder takes the encoded vectors (step 3) from the latent space and reconstructs the images (step 4). The AE adjusts its parameters to minimize the reconstruction loss, the difference between the originals and the reconstructions (step 5).

Follow the digit from image to latent vector to reconstruction, then trace how reconstruction loss updates the encoder and decoder. section 7.1.2

The encoding need not always be one deterministic point. A variational autoencoder (VAE) for color faces uses convolutional layers to reduce a 3 × 256 × 256 image to parameters of a 100-value latent distribution. It samples an encoding for the decoder; training combines reconstruction loss with KL divergence. Convolutions help handle these larger images without relying solely on dense layers. section 7.3.2 section 7.4.1

For translation, the input is a sequence rather than an image. Token embeddings receive positional encodings before entering the Transformer encoder. Each encoder layer applies multihead self-attention and a position-wise feed-forward network, with normalization and residual connections; six such layers are stacked in the implementation. The resulting vectors go to a decoder that predicts French tokens using the encoder output and previously generated tokens. An encoder-only Transformer instead supplies representations for downstream tasks without that decoder. section 9.1.1 section 9.2.2 section 1.3.2

Other encoder roles depend on what the next component needs:

• A denoising U-Net’s contracting path downsamples a noisy image and extracts features for its expansive path. section 15.1.2
• CLIP’s image and text encoders place paired inputs in a shared embedding space for comparison. section 15.5.1

How Learn Generative AI with PyTorch covers it

How deep it goes

Works through dense, convolutional, and stacked Transformer encoders with training steps, tensor dimensions, and layer code.

Examples

Uses handwritten-digit reconstruction, face-image generation, English-to-French translation, U-Net denoising, and CLIP image–text pairs.

What it assumes

Builds on word embeddings and earlier convolutional and residual-connection concepts while spelling out encoder inputs and layers.

What you can do after

Build it from scratch in PyTorch as part of digit autoencoders, face VAEs, and encoder–decoder Transformers.

How it connects

Links encoders to decoders, latent spaces, variational autoencoders, self-attention, positional encoding, and contrastive learning.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A sequence encoder is a neural-network component that maps an input sequence to contextualized representations for subsequent processing.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionA model or method that maps an input data object to a lower-dimensional learned representation.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe sequence model component that reads the source sequence and compresses it into a representation for another model component to use.
• 

  liveBookmanning.com
  Causal AIAn encoder is the component of a representation-learning model that maps observed inputs, optionally with labels or other conditioning variables, to a latent representation.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionAn encoder is a neural-network component that transforms high-resolution input representations into progressively more compact representations of higher-level information.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn encoder model transforms an input sequence into an intermediate representation that summarizes information from that sequence.
• 

  liveBookmanning.com
  Outlier Detection in PythonAn encoder is the component of an autoencoder that transforms an input into a compressed representation.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsAn encoder is a neural-network component that transforms each element of an input sequence into a context-dependent hidden representation.
• 

  liveBookmanning.com
  Generative AI in ActionA text encoder is a neural-network component that converts textual input into an internal vector representation.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe neural-network component that maps an input to a latent-space distribution.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsThe part of a sequence model or autoencoder that transforms an input sequence into an internal representation for downstream decoding.
• 

  liveBookmanning.com
  Transformers in ActionAn encoder is the sequence-to-sequence component that processes an input sequence and produces a representation for a decoder.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionAn encoder is a model component that transforms input data into a learned representation, such as vectors.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksAn encoder is a model component that transforms an input into a lower-dimensional or otherwise compressed representation.
• 

  liveBookmanning.com
  LLMs in ProductionAn encoder is the context-processing component of a transformer architecture that converts input tokens into contextual representations.
• 

  liveBookmanning.com
  Hugging Face in ActionA text encoder is a neural network component that converts textual input into an internal numerical representation.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsAn encoder is the sequence-processing component of a transformer that receives token representations and transforms them into contextual representations.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionAn encoder is a model component that transforms input data into a compressed or otherwise structured representation.
• 

  liveBookmanning.com
  Data Without LabelsAn encoder is a model component that maps an input into a lower-dimensional or otherwise transformed representation.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionAn encoder is the transformer component that transforms an input sequence into contextual representations through stacked attention and feedforward layers.

Compare how 21 books explain encoder →

Related concepts

• transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• embeddingproduces · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• autoencoderpart of · An autoencoder is a neural network that encodes input data into an internal representation and decodes it to reconstruct the input or an approximation.
• variational autoencoderpart of · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
• CLIPpart of · CLIP is a multimodal model and pretraining method that aligns image and text representations in a shared embedding space.
• U-Netpart of · U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.