liveBook · Manning

Tokenizer

As explained in Retrieval Augmented Generation, The Foundational Ideas

A tokenizer converts text into token identifiers and related tensors that a neural language model can process.

A tokenizer is the bridge between question-and-passage text and the tensors an encoder–decoder model can process. In Fusion-in-Decoder (FiD), each retrieved passage is paired with the question, and the resulting strings are tokenized as one batch before the passages are encoded independently. section 5.4

Tokenizing a passage batch

For each passage, format its title, context, and the question into an input string. Call the Hugging Face tokenizer with padding='longest', truncation=True, and return_tensors='pt'. Its input IDs go to the encoder; its attention mask accompanies the encoded representations into answer generation. The tokenizer also decodes generated IDs into readable text. section 5.4

Listing 5.2 Encoding for FiD.
from typing import List, Tuple
from typing import TypedDict
from transformers.modeling_outputs import BaseModelOutput
import torch
 
class Passage(TypedDict):  #A
  title: str
  text: str
 
def encode_passages(
  question: str,
  passages: List[Passage],
  tokenizer, model
) -> Tuple[torch.Tensor, torch.Tensor]:  #B
  prompt_template = (
    "Synthesize an answer for the question by combining "
    "all relevant information "
    "only from the following context.\n\n"
    "Title: {title}\n"
    "Context: {context}\n\n"
    "Question: {question}"
  )
  inputs = [
    prompt_template.format(
      title=p['title'],
      context=p['text'],
      question=question
    ) for p in passages
  ]
  input_encodings = tokenizer(
    inputs, padding='longest', truncation=True, return_tensors='pt'
  )
  input_ids = input_encodings.input_ids
  attention_mask = input_encodings.attention_mask
  encoder_outputs = model.encoder(  
    input_ids=input_ids,
    attention_mask=attention_mask,  
    return_dict=True 
  )
  return encoder_outputs.last_hidden_state, attention_mask
 

The encoding listing shows where batched text becomes input IDs and an attention mask. section 5.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Shows tokenizer calls and the resulting tensors in a worked FiD implementation, without explaining tokenization algorithms.
Examples
Uses a Hugging Face tokenizer with longest-batch padding and truncation for retrieved passages, plus decoding of generated IDs.
What it assumes
Assumes familiarity with Python functions and tensors while showing how to call the tokenizer for batched inputs.
What you can do after
Apply it via Hugging Face Transformers to prepare FiD encoder inputs and decode a generated answer.
How it connects
Links tokenization to attention masks, encoder hidden states, cross-attention, and encoder–decoder models.

Also explained in

Compare how 19 books explain tokenizer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.