A text chunk is a section of source material kept as a usable unit for retrieval. Semantic chunking aims to preserve meaningful content together, while token-count chunking provides a practical size boundary. section 6.3.2 section 11.1.4
In retrieval-augmented generation, chunking sits between document loading and embedding: documents become chunks, chunks become vectors, and retrieved chunks provide context for the language model. section 6.1.1
A practical chunking workflow is:
- Load the source document and divide it into manageable sections. section 6.3.2
- Keep chunks within a token limit, while favoring boundaries that preserve semantic coherence. section 6.3.2 section 11.1.4
- Embed each chunk and store the vectors for similarity retrieval. section 6.1.1
- Ground answers in retrieved chunks and acknowledge when the needed information is absent. section 6.3.2 section 11.1.4
Chunking a movie script
A simple implementation reads a movie script, accumulates words, and starts a new chunk when encoding the next word would exceed 200 tokens. The resulting chunks are embedded and indexed in Chroma DB for a script-search tool. section 6.3.2
Listing 6.6 02_RAG_agent_vector.py
script_text = Path("chapter_06/sample_documents/back_to_the_future.txt").read_text(
encoding="utf-8"
)
def simple_chunk(text, max_tokens=200): #1
tokenizer = tiktoken.get_encoding("cl100k_base")
words, chunk, chunks = text.split(), [], []
for w in words:
if len(tokenizer.encode(" ".join(chunk + [w]))) > max_tokens:
chunks.append(" ".join(chunk))
chunk = [w]
else:
chunk.append(w)
if chunk:
chunks.append(" ".join(chunk))
return chunks
docs = simple_chunk(script_text, max_tokens=200)
client = chromadb.PersistentClient( #2
path="./chapter_06/chroma_script_store"
)
collection_name = "bttf_script"
try:
collection = client.get_collection(collection_name) #3
except Exception: #4
collection = client.create_collection(name=collection_name) #4
if collection.count() == 0: #4
collection.add(ids=[str(uuid.uuid4()) for _ in docs], documents=docs)
@function_tool #5
def search_script(query: str, top_k: int = 3) -> str:
res = collection.query(query_texts=[query], n_results=top_k)
if res and "documents" in res and res["documents"] and res["documents"][0]:
return "\n\n".join(res["documents"][0])
return "No relevant documents found."
agent = Agent(
name="Script Agent",
instructions=( #6
"You answer questions about the movie *Back to the Future*.\n" #7
"When needed, call the `search_script` tool to fetch passages, " #7
"then cite or paraphrase them in your answer." #7
"Make sure your answers are grounded in the script text.\n" #7
), #7
tools=[search_script],
)
query = "Where does Doc tell Marty to meet him, and at what time?"
result = Runner.run_sync(agent, query)
print("\n--- ANSWER ---\n", result.final_output)
query = "What happens at 1:15AM" #7
result = Runner.run_sync(agent, query)
print("\n--- ANSWER ---\n", result.final_output)
OUTPUT
--- ANSWER ---
Doc tells Marty to meet him at the Twin Pines Mall parking lot at 1:15 AM.
--- ANSWER ---
There's no specific event or mention of 1:15 AM in the script passages I retrieved. If you have a particular scene or context in mind, let me know, and I can help further! #7
Inspect the complete token-limited chunking function and the surrounding vector-store setup. section 6.3.2
