liveBook · Manning

DirectoryLoader

As explained in Build Python Web Apps with Streamlit

DirectoryLoader is a LangChain loader that discovers and loads files from a specified directory.

DirectoryLoader discovers files in a directory and loads them into LangChain Document objects, making folder-based knowledge bases easier to ingest. section 10.5.2

Load text files from a folder

Point the loader at an articles/ folder, include .txt files in nested folders, and use TextLoader for each file:\n\nloader = DirectoryLoader(\n folder_path,\n glob="**/*.txt",\n loader_cls=TextLoader\n)\ndocuments = loader.load()\n\nThe result is a list of Documents containing raw text and metadata. section 10.5.2

How Build Python Web Apps with Streamlit covers it

How deep it goes
Defines glob selection, delegated file loading, and Document output, then defers chunking to a separate splitter.
Examples
Uses an articles/ folder, the **/*.txt pattern, TextLoader, and a loader.load() call.
What it assumes
Assumes a LangChain application and a folder of text files, while demonstrating the loader configuration directly.
What you can do after
Apply it via LangChain to turn a directory of text files into documents ready for splitting and vector storage.
How it connects
Connects TextLoader, Document objects, RecursiveCharacterTextSplitter, vector stores, and retrieval.
Cover of Build Python Web Apps with Streamlit
liveBookmanning.com
Read Build Python Web Apps with Streamlit on liveBook Get the book at manning.com

Also explained in

Compare how 2 books explain DirectoryLoader →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build Python Web Apps with Streamlit. The section links point to the parts of the book it draws on.