Document chunking divides large documents into smaller Documents before embedding and storage, preventing an entire document from being sent as prompt context. section 4.3.3
A typical Spring AI loading pipeline connects these steps:
- Read a file or uploaded byte stream into a Document. section 4.3 section 4.3.3
- Apply TokenTextSplitter to produce a List<Document> containing the chunks. section 4.3 section 4.3.3
- Pass the resulting chunks to a vector store for embedding-backed retrieval. section 4.1 section 4.3
TokenTextSplitter targets a maximum of 800 tokens by default and avoids splitting mid-sentence. Its builder can adjust chunk size, separator retention, maximum chunk count, and minimum lengths. section 4.3.3
Configuring a splitter
TextSplitter splitter = TokenTextSplitter.builder()\n .withChunkSize(500)\n .withKeepSeparator(false)\n .withMinChunkLengthToEmbed(10)\n .build(); section 4.3.3
@Value("file://${HOME}/documents/my-document.txt")
private Resource documentResource;
public void loadDocument(VectorStore vectorStore) {
DocumentReader reader = new TextReader(documentResource);
TextSplitter splitter = TokenTextSplitter.builder().build();
vectorStore.accept(splitter.apply(reader.get()));
}
See the smallest working pipeline: TextReader loads a resource, TokenTextSplitter creates chunks, and VectorStore accepts them. section 4.3
