A BART encoder converts a tokenized text caption or prompt into dense vector embeddings. In min-DALL-E, these embeddings condition a BART decoder that predicts image tokens for a VQGAN decoder. section 12.1.1 section 12.5
The prompt-to-embedding path proceeds through these implementation stages:
- Download the tokenizer vocabulary and pretrained encoder weights. section 12.2 section 12.2.2
- Tokenize, pad, and format the prompt as model input. section 12.2
- Instantiate DalleBartEncoder, load its pretrained parameters, and run the encoder in evaluation mode. section 12.2.2
- Pass the resulting embeddings to the autoregressive BART decoder, which generates image tokens that the VQGAN decoder reconstructs into an image. section 12.1.2
Encoding a concrete prompt
For “panda with top hat reading a book,” the encoder represents each of 64 text positions with a 2,048-dimensional vector, producing an embedding tensor shaped [2, 64, 2048] when unconditional and conditional prompts are encoded together. section 12.2 section 12.2.2
Listing 12.5 Initializing the pretrained BART encoder
from min_dalle.models import DalleBartEncoder
torch.manual_seed(42) #1
dtype = "float16"
print("initializing DalleBartEncoder")
encoder = DalleBartEncoder( #2
attention_head_count = 32,
embed_count = 2048, #3
glu_embed_count = 4096,
text_token_count = 64, #4
text_vocab_count = 50272,
layer_count = 24,
device=device
).to(getattr(torch, dtype)).eval()
params = torch.load(encoder_path)
encoder.load_state_dict(params, strict=False) #5
del params
encoder = encoder.to(device)
Inspect the PyTorch initialization showing attention heads, embedding sizes, vocabulary size, layers, precision, and pretrained parameter loading. section 12.2.2
