T5ForConditionalGeneration is a pretrained Hugging Face T5 model class for conditional text generation. It can serve as the encoder-decoder model in Fusion-in-Decoder (FiD): each retrieved passage is encoded with the question, then the decoder generates an answer from the combined representations. The runnable example loads a Flan-T5 model through AutoModelForSeq2SeqLM rather than naming T5ForConditionalGeneration directly. section 5.4
Listing 5.2 Encoding for FiD.
from typing import List, Tuple
from typing import TypedDict
from transformers.modeling_outputs import BaseModelOutput
import torch
class Passage(TypedDict): #A
title: str
text: str
def encode_passages(
question: str,
passages: List[Passage],
tokenizer, model
) -> Tuple[torch.Tensor, torch.Tensor]: #B
prompt_template = (
"Synthesize an answer for the question by combining "
"all relevant information "
"only from the following context.\n\n"
"Title: {title}\n"
"Context: {context}\n\n"
"Question: {question}"
)
inputs = [
prompt_template.format(
title=p['title'],
context=p['text'],
question=question
) for p in passages
]
input_encodings = tokenizer(
inputs, padding='longest', truncation=True, return_tensors='pt'
)
input_ids = input_encodings.input_ids
attention_mask = input_encodings.attention_mask
encoder_outputs = model.encoder(
input_ids=input_ids,
attention_mask=attention_mask,
return_dict=True
)
return encoder_outputs.last_hidden_state, attention_mask
The encoding function pairs one question with each passage, tokenizes those inputs, and obtains encoder hidden states and an attention mask. section 5.4
Listing 5.3 Decoding for FiD.
def decode_answer(encoder_hidden_states, attention_mask, tokenizer, model):
num_passages, seq_len, hidden_dim = encoder_hidden_states.shape
fused_hidden_states = encoder_hidden_states.reshape(
1, num_passages * seq_len, hidden_dim #A
)
fused_attention_mask = attention_mask.reshape(1, num_passages * seq_len)
encoder_outputs = BaseModelOutput(last_hidden_state=fused_hidden_states)
generated_ids = model.generate( #B
encoder_outputs=encoder_outputs,
attention_mask=fused_attention_mask,
max_length=64 #C
)
return tokenizer.decode(generated_ids[0], skip_special_tokens=True)The decoding function reshapes passage states and their mask into a single decoder memory, then passes them to generate. This fusion step must be implemented explicitly. section 5.4
