A text chunk is a smaller piece of a document prepared for an LLM workflow. Splitting a long document into chunks helps its content fit within a model’s context window while retaining enough surrounding meaning for summarization or retrieval. A chunk can be carried as a LangChain.js Document, with text and metadata ready for embedding or vector storage. section 6.2.1 section 7.1.3
A splitter first separates text into smaller meaningful units, then combines them up to a size limit. Overlap repeats some content across adjacent chunks so a boundary does not abruptly remove context. RecursiveCharacterTextSplitter works through separators from larger structures, such as newlines, toward smaller ones; alternatives split by tokens or according to HTML, Markdown, or code structure. section 6.2.1
Table 6.2 LangChain text splitters
| Name |
Splits |
Adds metadata? |
Description/use case |
| Recursive |
Specific character |
Yes |
Recursively splits text while keeping related pieces together; recommended for initial splitting |
| HTML |
HTML characters |
Yes |
Splits text based on HTML tags and structures |
| Markdown |
Markdown syntax |
Yes |
Splits text according to Markdown syntax |
| Code |
Code-specific characters |
Yes |
Splits code based on syntax specific to various programming languages |
| Token |
A token |
No |
Splits text on tokens, with various methods available for token measurement |
| Character |
A single user-defined character |
No |
Simple splitting based on a specified character |
Compare splitters by what they split and whether they add metadata before choosing one for a document format. section 6.2.1
Split loaded documents for summarization
After loading a PDF or DOCX into LangChain.js Documents, split the parsed output: const splitter = new RecursiveCharacterTextSplitter({ chunkSize: 1500, chunkOverlap: 200, }); const documentChunks = await splitter.splitDocuments(documents); Here, chunkSize and chunkOverlap are character counts. The resulting chunks can be passed into a summarization workflow; the chosen sizes do not themselves establish a model’s token count. section 7.1.3 section 6.2.1
Chunks determine what later stages receive. In MapReduce summarization, each chunk is summarized independently before those summaries are combined; refine instead updates a running summary as subsequent chunks arrive. For retrieval, chunks can become embeddings stored in a vector database. section 7.1.2 section 6.2.1
Treat chunk quality and processing cost as implementation concerns. PDFs may lack reliable headings or sections, making meaningful boundaries harder to find than in DOCX files. For very large documents, looping over every chunk can still be slow or expensive; embedding and clustering similar chunks is one explored way to assemble more coherent sections. In an upload pipeline, validate files and keep parsing and chunking on the server with error handling. section 7.1.6 section 11.2.2
