A document loader reads source data and converts it into structured Document objects for downstream processing. In a retrieval-augmented generation workflow, it turns files into material that can be split, embedded, and stored for search. section 10.5.2
Loading text files from a folder
DirectoryLoader scans a folder, while TextLoader handles each matching text file. With glob="**/*.txt", loader.load() returns a list of Document objects, including the files' raw text and metadata. A text splitter can then divide long articles into smaller chunks before vector-store ingestion.
A practical ingestion sequence is:
- Choose a loader suited to the source, such as TextLoader for individual text files. section 10.5.2
