A tokenizer is the bridge between question-and-passage text and the tensors an encoder–decoder model can process. In Fusion-in-Decoder (FiD), each retrieved passage is paired with the question, and the resulting strings are tokenized as one batch before the passages are encoded independently. section 5.4
Tokenizing a passage batch
For each passage, format its title, context, and the question into an input string. Call the Hugging Face tokenizer with padding='longest', truncation=True, and return_tensors='pt'. Its input IDs go to the encoder; its attention mask accompanies the encoded representations into answer generation. The tokenizer also decodes generated IDs into readable text. section 5.4
Listing 5.2 Encoding for FiD.
from typing import List, Tuple
from typing import TypedDict
from transformers.modeling_outputs import BaseModelOutput
import torch
class Passage(TypedDict): #A
title: str
text: str
def encode_passages(
question: str,
passages: List[Passage],
tokenizer, model
) -> Tuple[torch.Tensor, torch.Tensor]: #B
prompt_template = (
"Synthesize an answer for the question by combining "
"all relevant information "
"only from the following context.\n\n"
"Title: {title}\n"
"Context: {context}\n\n"
"Question: {question}"
)
inputs = [
prompt_template.format(
title=p['title'],
context=p['text'],
question=question
) for p in passages
]
input_encodings = tokenizer(
inputs, padding='longest', truncation=True, return_tensors='pt'
)
input_ids = input_encodings.input_ids
attention_mask = input_encodings.attention_mask
encoder_outputs = model.encoder(
input_ids=input_ids,
attention_mask=attention_mask,
return_dict=True
)
return encoder_outputs.last_hidden_state, attention_mask
The encoding listing shows where batched text becomes input IDs and an attention mask. section 5.4


















