liveBook · Manning

T5ForConditionalGeneration

As explained in Retrieval Augmented Generation, The Foundational Ideas

T5ForConditionalGeneration is a Hugging Face Transformers model class that performs conditional text generation with a T5 encoder-decoder model.

T5ForConditionalGeneration is a pretrained Hugging Face T5 model class for conditional text generation. It can serve as the encoder-decoder model in Fusion-in-Decoder (FiD): each retrieved passage is encoded with the question, then the decoder generates an answer from the combined representations. The runnable example loads a Flan-T5 model through AutoModelForSeq2SeqLM rather than naming T5ForConditionalGeneration directly. section 5.4

Listing 5.2 Encoding for FiD.
from typing import List, Tuple
from typing import TypedDict
from transformers.modeling_outputs import BaseModelOutput
import torch
 
class Passage(TypedDict):  #A
  title: str
  text: str
 
def encode_passages(
  question: str,
  passages: List[Passage],
  tokenizer, model
) -> Tuple[torch.Tensor, torch.Tensor]:  #B
  prompt_template = (
    "Synthesize an answer for the question by combining "
    "all relevant information "
    "only from the following context.\n\n"
    "Title: {title}\n"
    "Context: {context}\n\n"
    "Question: {question}"
  )
  inputs = [
    prompt_template.format(
      title=p['title'],
      context=p['text'],
      question=question
    ) for p in passages
  ]
  input_encodings = tokenizer(
    inputs, padding='longest', truncation=True, return_tensors='pt'
  )
  input_ids = input_encodings.input_ids
  attention_mask = input_encodings.attention_mask
  encoder_outputs = model.encoder(  
    input_ids=input_ids,
    attention_mask=attention_mask,  
    return_dict=True 
  )
  return encoder_outputs.last_hidden_state, attention_mask
 

The encoding function pairs one question with each passage, tokenizes those inputs, and obtains encoder hidden states and an attention mask. section 5.4

Listing 5.3 Decoding for FiD.
def decode_answer(encoder_hidden_states, attention_mask, tokenizer, model):
 
    num_passages, seq_len, hidden_dim = encoder_hidden_states.shape
 
    fused_hidden_states = encoder_hidden_states.reshape(
        1, num_passages * seq_len, hidden_dim  #A
    )
    fused_attention_mask = attention_mask.reshape(1, num_passages * seq_len)
 
    encoder_outputs = BaseModelOutput(last_hidden_state=fused_hidden_states)
 
    generated_ids = model.generate(  #B
        encoder_outputs=encoder_outputs,
        attention_mask=fused_attention_mask,
        max_length=64  #C
    )
 
    return tokenizer.decode(generated_ids[0], skip_special_tokens=True)

The decoding function reshapes passage states and their mask into a single decoder memory, then passes them to generate. This fusion step must be implemented explicitly. section 5.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Works through passage encoding, tensor reshaping, generation, and a fine-tuning step; does not implement the T5 architecture itself.
Examples
Uses a multi-passage project question, Flan-T5 loading code, generated answer, and fine-tuning code.
What it assumes
Introduces the encoder-decoder mechanism while relying on Python, tokenizers, and tensor operations in the implementation.
What you can do after
Apply it via Hugging Face Transformers to encode passages separately and generate an answer from fused encoder outputs.
How it connects
Connects encoder-decoder architecture and cross-attention to retrieval-augmented generation and Fusion-in-Decoder.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.