A document loader brings external content into an LLM workflow. In LangChain, a source-specific loader extracts content from a file, web page, SaaS tool, or database and turns it into standardized Document objects. Each Document holds content and related metadata, giving later components a consistent input. section 1.2.1 section 1.7
For a retrieval-based Q&A application, ingestion typically proceeds as follows:
- Choose a loader for the source. For CSV data, CsvLoader is one example; it parses the source into Document objects. section 7.1.1
- Pass large Documents to a TextSplitter, which produces smaller Document chunks. An embedding model converts chunk text to vectors. section 7.1.1
- Store chunks and embeddings in a vector store so a retriever can later find relevant context for an answer. section 7.1.1 section 1.2.1
Loading for summarization
To summarize content drawn from a Wikipedia page and local TXT, DOCX, and PDF files, select an appropriate DocumentLoader for each source. If the input is already a set of Documents, it need not be split merely to create the list; a large document may still need chunking. section 3.2.1
Keep the responsibilities separate: the loader imports content; it does not itself retrieve answers. This separation lets an application change its data connector without rebuilding the rest of the ingestion workflow. section 1.2 section 1.2.1
Figure 7.2 Content ingestion sequence diagram
Follow the ingestion sequence to see where loading ends and splitting, embedding, and storage begin. section 7.1.1
