A text chunk is a smaller piece of a document prepared for retrieval. In a retrieval-augmented generation (RAG) pipeline, recipes are split into chunks before their text is converted to vectors. Keeping each chunk tied to its recipe lets a retrieved result point back to its source. section 14.5 section 14.11
Chunk size is a practical trade-off. Putting thousands of recipes in one chunk makes the relevant source hard to pinpoint. Splitting every recipe into 30–50 sentence-sized chunks can return many results from the same source. A useful split here is two chunks per recipe: one for ingredients and one for instructions. section 14.5
Two rows from one recipe
For each recipe file, create two pandas DataFrame rows. Each row carries the recipe name and a chunk containing that name plus either the ingredients or the instructions. The name preserves source context in both pieces. section 14.5
Listing 14.3 Splitting recipes into chunks
from typing import List
import pandas as pd
def create_chunks(files_to_process: List[str], path:str) -> pd.DataFrame:
chunks = []
for file in files_to_process:
recipe_name = file.replace(".json","").split("/")[-1]
data = load_json_from_minio(recipe_name, path)
ingredients = data.get('ingredients', 'No ingredients found')
instructions = data.get('instructions', 'No instructions found')
chunks.append({
"recipe_name": recipe_name,
"chunk": f"{recipe_name} \n Ingredients: {ingredients}" #1
})
chunks.append({
"recipe_name": recipe_name,
"chunk": f"{recipe_name} \n Instructions: {instructions}" #2
})
return pd.DataFrame(chunks)
The create_chunks function loads each recipe, appends its ingredient and instruction chunks, and returns the rows as a DataFrame. section 14.5
In the scheduled ingestion workflow, a DockerOperator runs preprocessing for a date-specific path. Preprocessing finds that day’s JSON recipes, creates chunks, assigns recipe and chunk UUIDs, and saves the resulting DataFrame for later vector-database work. section 14.5 section 14.11
