liveBook · Manning

Collection

As explained in Data Pipelines with Apache Airflow, Second Edition

A collection is a named group of related objects with defined properties and vectorization settings in Weaviate.

A collection is a named group of related objects in Weaviate, comparable to a table in a traditional database. It defines the fields stored with each object and the vectorizer configuration used to turn suitable text into embeddings. In a recipe-ingestion pipeline, a dedicated Airflow task ensures that the collection exists before recipes are stored. section 14.6 section 14.11

Creating a collection requires three coordinated choices:

  • Choose a collection name, such as recipes, to identify the records logically. section 14.6
  • Define properties as named fields with Weaviate data types, such as UUID for identifiers and TEXT for content. section 14.6.2
  • Configure the vectorizer and embedding model that determine how vectorized text connects to the embedding service. section 14.6

Recipe collection schema

A recipe collection can contain recipe_uuid, recipe_name, chunk_uuid, and chunk. UUID identifiers can skip vectorization, while the recipe name and content chunk remain TEXT fields eligible for vectorization. The chunk stores the portion of recipe content used for retrieval. section 14.6.2

Collection creation should be repeatable without recreating an existing collection. Check the Weaviate instance first; if the name already exists, log the condition and return. Otherwise, create the collection with its vectorizer configuration and properties. Changing the schema requires deleting the existing collection before running creation again, so the dedicated Airflow task also provides a visible operational checkpoint. section 14.6.3

How Data Pipelines with Apache Airflow, Second Edition covers it

How deep it goes
Explains naming, properties, data types, vectorizers, existence checks, and schema replacement, but not Weaviate internals.
Examples
Uses a recipes collection with UUID identifiers, recipe text, content chunks, and a DockerOperator creation task.
What it assumes
Assumes familiarity with Weaviate, Airflow tasks, vector databases, and embeddings while introducing collection schemas from the table analogy.
What you can do after
Operate/configure a Weaviate collection for recipe ingestion with typed properties, vectorization settings, and repeatable creation checks.
How it connects
Connects collections with vector databases, schemas, properties, embeddings, vectorization, RAG, and Airflow tasks.

Also explained in

Compare how 16 books explain collection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Pipelines with Apache Airflow, Second Edition. The section links point to the parts of the book it draws on.