liveBook · Manning

Tokenization

As explained in Build an Advanced RAG Application (From Scratch)

Tokenization is the process of splitting text into words, phrases, or other meaningful units for language processing.

Tokenization splits text into words, phrases, subwords, sentences, or other meaningful units. In keyword retrieval, these tokens become terms that an inverted index maps to document IDs for fast lookup. section 3.2 section 4.2

Build a basic tokenization-and-indexing pipeline by:

  • Starting with searchable documents. section 3.2
  • Splitting each document into tokens such as individual words. section 3.2
  • Creating a mapping from each token to the document IDs containing it. section 3.2
  • Appending document IDs while iterating over tokens, then retrieving matching documents directly from a token’s list. section 3.2

Two tokenization levels

The sentence “The quick brown fox jumps over the lazy dog” becomes individual word tokens. Sentence-level tokenization instead treats a complete sentence as one unit, enabling a model to produce one fixed-size embedding for the sentence. section 4.2 section 4.2

Listing 4.1 Generate Embeddings on sample data
from sentence_transformers import SentenceTransformer #A
 
model = SentenceTransformer(“all-MiniLM-L6-v2”) #B
 
sentences = ["This is an example sentence.", "Another example sentence."]
 
embeddings = model.encode(sentences) #C
 
print(len(embeddings)) #D

A runnable Sentence Transformers listing connects tokenization to encoding sentences as embedding vectors. section 4.2

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains word, subword, and sentence units, then shows token iteration and inverted-index population without detailing tokenizer internals.
Examples
Uses machine-learning documents, a fox sentence, and sample sentences encoded with SentenceTransformer.
What it assumes
Assumes text documents and retrieval goals while teaching word-level, subword-level, and sentence-level splitting directly.
What you can do after
Apply it via an inverted index or Sentence Transformers to prepare text for keyword lookup or semantic retrieval.
How it connects
Connects tokenization to inverted indexes, TF-IDF, embeddings, semantic search, and Sentence Transformers.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.