IVFFlat is an approximate nearest-neighbor index for vector embeddings. It groups similar embeddings into inverted lists, each organized around a cluster centroid, so searches inspect selected groups instead of the entire dataset. section 8.5 section 8.5.1
IVFFlat search proceeds in three practical steps:
- Computes centroids with k-means clustering and assigns each embedding to its nearest centroid. section 8.5.1
- Compares the query embedding with centroids, then searches the list whose centroid is closest. section 8.5.1
- Trades exactness for speed because it compares against only a subset of stored embeddings; probes can increase recall. section 8.5 section 8.5.1
Create and inspect an IVFFlat index
CREATE INDEX movie_embeddings_ivfflat_idx ON omdb.movies USING ivfflat (movie_embedding vector_cosine_ops) WITH (lists = 5); A similarity query using the same cosine-distance operator can use this index. EXPLAIN reports an Index Scan and approximately 1.5 ms execution time, compared with a sequential scan taking approximately 52.6 ms. section 8.5.1
Listing 8.9 Creating an IVFFlat index
CREATE INDEX movie_embeddings_ivfflat_idx ON omdb.movies USING ivfflat (movie_embedding vector_cosine_ops) WITH (lists = 5);
Shows the executable CREATE INDEX statement for a movie-embedding column. section 8.5.1
Listing 8.11 Increasing the number of probes for an IVFFlat index
BEGIN;
SET LOCAL ivfflat.probes = 2;
SELECT id, name
FROM omdb.movies
ORDER BY movie_embedding <=>
(SELECT phrase_embedding
FROM omdb.phrases_dictionary
WHERE phrase = 'May the force be with you')
LIMIT 3;
COMMIT;
Shows how to set ivfflat.probes locally before running a similarity query. section 8.5.1
