liveBook · Manning

TokenTextSplitter

As explained in Spring AI in Action

TokenTextSplitter is a Spring AI document splitter that divides text into token-bounded chunks while avoiding mid-sentence splits.

TokenTextSplitter is a Spring AI TextSplitter that converts large Documents into smaller, token-bounded Documents before they are stored in a vector store. It supports document-ingestion pipelines for retrieval-augmented applications and avoids splitting text in the middle of a sentence. section 4.3 section 4.3.3

Configure the splitter through its builder when the default behavior does not fit the ingestion workload:

  • Target chunks contain no more than 800 tokens by default; smaller chunks may result when sentence boundaries require them. section 4.3.3

A configured splitter

TextSplitter splitter = TokenTextSplitter.builder() .withChunkSize(500) .withKeepSeparator(false) .withMinChunkLengthToEmbed(10) .build();

How Spring AI in Action covers it

How deep it goes
Explains token limits, sentence preservation, defaults, and builder settings, but does not detail the splitting algorithm internals.
Examples
Uses a text-file loading flow, a reactive TikaDocumentReader pipeline, and a 500-token builder configuration.
What it assumes
Assumes familiarity with Documents, document readers, vector stores, and basic Spring bean or reactive pipeline configuration.
What you can do after
Apply it via Spring AI to split documents before writing their chunks to a vector store.
How it connects
Connects DocumentReader, TextSplitter, Document, VectorStore, and reactive Flux pipelines.
Cover of Spring AI in Action
liveBookmanning.com
Read Spring AI in Action on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Spring AI in Action. The section links point to the parts of the book it draws on.