Parsing turns files into clean text while preserving useful structure. This matters because PDFs can scramble columns or tables, Word files can carry revision history, and HTML can mix article content with navigation and boilerplate. A shared ingestion pipeline lets teams reuse fixes instead of maintaining separate extractors. section 5.3.1
A pipeline detects a file’s format, routes it to a matching parser, then passes extracted text onward for metadata capture and chunking. Parsers return sections with content, headings, and hierarchy; callers can derive one text string when needed. Detection checks file headers before falling back to extensions, with plain text as the fallback. section 5.3.2 section 5.3.3
Choose a parser to match the source format and its structure:
- PDF parsers may need to handle multi-column layouts, tables, and repeated headers. section 5.3.3
- Markdown parsers can use explicit headings, paragraph breaks, and code blocks; HTML parsers need to remove navigation and other boilerplate. section 5.3.3
- Plain-text parsers can return a single section without a heading. section 5.3.3
Listing 5.4 Document parser interface and extracted document model
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List, Optional, Dict
@dataclass
class DocumentSection:
content: str
heading: Optional[str] = None #A
level: int = 0 #A
page_number: Optional[int] = None
@dataclass
class ExtractedDocument:
sections: List[DocumentSection] #B
metadata: Dict[str, str] #C
@property
def text(self) -> str: #D
return "\n\n".join(s.content for s in self.sections)
class DocumentParser(ABC):
@abstractmethod
def parse(self, file_bytes: bytes,
filename: str) -> ExtractedDocument: #E
"""Extract text and structure from a document."""
passThe parser interface and extracted-document model show how format-specific implementations return shared structural data. section 5.3.3
Keep the pipeline extensible: implement a parser against the shared interface and register it for a new format. The rest of the pipeline can continue with chunking, embedding, and storage unchanged. section 5.3.3
















