Document clustering organizes large documents by semantic similarity, preparing coherent sections for more efficient summarization. The workflow splits text into chunks, embeds each chunk as a vector, applies k-means, and merges related clusters. section 7.1.6
A production-oriented clustering pipeline includes:
- Split documents with RecursiveCharacterTextSplitter using 10,000-character chunks and 250-character overlap. section 7.1.6
- Generate dense chunk embeddings with the Xenova/all-MiniLM-L6-v2 feature-extraction pipeline. section 7.1.6
- Choose a cluster count with a square-root heuristic, then merge semantically related clusters using a similarity threshold. section 7.1.6
JavaScript implementation
A clusterDocuments function can orchestrate embedding computation, k-means clustering, empty-input validation, heuristic cluster selection, and mergeSimilarClusters processing with ml-kmeans and Hugging Face Transformers.
