Weaviate is an open-source vector database for storing objects alongside vector representations and supporting semantic search in machine-learning applications. It organizes related objects into collections, comparable to tables whose fields describe business entities. section 14.11 section 14.6
A recipe-ingestion pipeline prepares JSON data, adds UUIDs, creates a pandas DataFrame, and uploads processed records to MinIO before loading them into Weaviate. The collection defines the recipe name, properties, and vectorizer configuration. section 14.10 section 14.6
A repeatable ingestion workflow handles both initial loading and later changes:
- Create a recipes collection with fields and an embedding-model configuration. section 14.6
- Compare incoming objects with existing records and mark changed recipes for update. section 14.10
- Delete outdated records, then upload new and updated recipes to the vector database. section 14.10
- Use stored vectors within a retrieval-augmented generation workflow to produce recipe recommendations. section 14.10 section 14.11
Create a recipe collection
A pipeline task can invoke the Python package CLI with a collection name and embedding model: command=f"create {COLLECTION_NAME} text-embedding-3-large". The underlying client creates the collection with a name, vectorizer_config, and properties.
Figure 14.15 Upload our recipes to the vector database using the 01_ingestion DAG.
Trace the ingestion DAG from prepared recipe files through comparison, deletion, and upload into the vector database. section 14.10
