Document loader
A document loader is a software component that reads content from a source and converts it into a standardized document representation for downstream processing.
Build Python Web Apps with Streamlit
A document loader is a component that reads source data and converts it into structured documents for downstream processing.
- How it is used
- Supports RAG ingestion by converting files into searchable structured documents.
- How deep it goes
- Explains loader selection and folder loading, but does not detail parsing internals for formats beyond text files.
- Examples
- Uses DirectoryLoader, TextLoader, a recursive text splitter, and an articles folder containing text files.
- What you can do after
- Build it from scratch by loading a folder of text files and preparing Document objects for vector-store ingestion.
Read how Build Python Web Apps with Streamlit explains document loader →
AI Agents and Applications
A data loader is a component that ingests source data into a form suitable for downstream language-model processing.
- How it is used
- Imports external content as standardized Documents for summarization or retrieval-based question answering.
- How deep it goes
- Defines the loader’s input and output and walks through its place in ingestion, without detailing loader implementation.
- Examples
- Uses a CSV ingestion flow and summarization inputs from Wikipedia plus local TXT, DOCX, and PDF files.
- What you can do after
- Apply it via LangChain’s source-specific DocumentLoader to prepare content for summarization or a retrieval pipeline.
Read how AI Agents and Applications explains document loader →
Effective Conversational AI
A document loader is a component that reads content from a configured source or file format and represents it as documents with text and metadata.
- How it is used
- Starts retrieval-augmented generation ingestion by converting diverse source files into documents for downstream processing.
- How deep it goes
- Defines loader responsibilities and names common implementations, but defers detailed parsing, chunking, and transformation internals.
- Examples
- Uses text files, CSV files, directories, PDFs, HTML, markdown, webpages, transcripts, and Microsoft Office documents.
- What you can do after
- Operate LangChain loaders to ingest configured files and hand structured documents to retrieval pipelines.
Read how Effective Conversational AI explains document loader →
Related concepts
- Documentproduces · A discrete representation of information, such as content or a structured record, optionally accompanied by metadata.
- vector databaseuses · A vector database is a data-management system that stores and indexes numerical vector representations and supports retrieval through similarity or distance searches.
- PyPDFLoaderhas part · PyPDFLoader is a LangChain document-loader class that reads PDF files, extracts their contents, and produces document objects.
- LangChainused by · LangChain is an open-source software framework for building applications that compose language models with prompts, data sources, tools, and other components.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.