liveBook · Manning

DirectoryLoader

As explained in Machine Learning Platform Engineering

DirectoryLoader is a document-loading component that recursively discovers files in directories and converts them into standardized document objects.

DirectoryLoader supplies the document-loading stage of a retrieval-augmented generation pipeline. It recursively traverses a documentation directory, selects matching files, and returns standardized Document objects with content and metadata such as file paths and modification dates. section 12.2.4

RAG indexing pipeline

load_documents(data_dir) feeds documents into RecursiveCharacterTextSplitter, then a FAISS index. A similarity search can validate the resulting index with an operational question such as “How do I rotate encryption keys?”

How Machine Learning Platform Engineering covers it

How deep it goes
Explains recursive traversal, file matching, loader selection, encoding, and loading, then hands documents to chunking.
Examples
Uses nested Markdown documentation, TextLoader, UTF-8 decoding, and a FAISS indexing pipeline as concrete examples.
What it assumes
Assumes Python functions and directories while demonstrating the loader configuration from the document-loading stage.
What you can do after
Build it from scratch by loading selected files into standardized documents for chunking and vector indexing.
How it connects
Connects document loading with metadata, document chunking, embeddings, vector indexes, and retrieval-augmented generation.
Cover of Machine Learning Platform Engineering
liveBookmanning.com
Read Machine Learning Platform Engineering on liveBook Get the book at manning.com

Also explained in

Compare how 2 books explain DirectoryLoader →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Platform Engineering. The section links point to the parts of the book it draws on.