An approximate nearest neighbor algorithm finds nearby points without computing exact nearest-neighbor relationships exhaustively. It trades some exactness for faster distance searches, making it useful when outlier detection or related workflows must repeatedly process large numeric datasets. section 10.2.5
Practical considerations shape the choice of an approximate method:
- Use approximate searches when pairwise distance calculations become a performance constraint. section 10.2.5
- Expect stronger support for numeric distances than categorical distances. section 10.2.5
- Treat the result as an input to outlier detection or clustering rather than as an automatically meaningful finding. section 10.2.5
HNSW nearest-neighbor query
Create 1,000 random records with 10 features, insert each record into a datasketch HNSW index using Euclidean distance, and query the first record for its 10 nearest neighbors. section 10.2.5
Listing 10.7 Example using HNSW for nearest neighbors queries
from datasketch.hnsw import HNSW
import numpy as np
data = np.random.random_sample((1000, 10)) #1
index = HNSW(distance_func=lambda x, y: np.linalg.norm(x - y))
for i, d in enumerate(data): #2
index.insert(i, d)
index.query(data[0], k=10) #3
The listing shows the complete HNSW workflow: create numeric data, define a distance function, insert records, and query nearest neighbors. section 10.2.5
