Suppose you want to answer questions about a PDF or summarize a Wikipedia article. The first step is to turn the source text into something the rest of the application can handle consistently: a LangChain Document. section 3.2.1 section 7.1.1
A Document packages text with metadata such as its source, page number, or timestamp. The text supplies content for search, answers, or summaries; the metadata helps preserve where that content came from as it moves through a pipeline. section 3.4 section 6.1.1
Figure 1.7 Object model of classes associated with the Document
core entity, including Document loaders (create Document objects), splitters (create a
list of Document objects), vector stores (store Document objects in vector stores),
and retrievers (retrieve Document objects from vector stores and other sources)
The object-model diagram places Document at the center: loaders create it, splitters produce smaller Documents, vector stores hold them, and retrievers bring them back. section 1.2.2
A typical ingestion path is:
- Load external text, such as a CSV file, into a Document. section 7.1.1
- Split a large Document into a list of smaller Documents when needed. If the inputs are already separate, splitting may be unnecessary. section 7.1.1 section 3.2.1
- Embed the chunks and store them for retrieval. A vector store may compute the embeddings itself when configured with an embedding model. section 7.1.1
Load Wikipedia content
With the loader package installed, use `WikipediaLoader(query="Paestum", load_max_docs=2).load()` to obtain a list of Documents. Do not assume that one requested article means one returned Document: linked Wikipedia pages may also be loaded. The limit of two keeps this summarization example’s cost down. section 3.2.2
Documents also provide the units for summarizing across sources. A MapReduce approach summarizes them independently before combining the results; a refine approach incorporates each one into a running summary. Choose with the trade-off in mind: parallel speed versus preserving more context. section 3.2 section 3.4
