Format detection identifies an input file before parsing so an ingestion pipeline can route it to the parser responsible for that format. The parser then produces clean text while preserving structure for downstream chunking. section 5.3.2 section 5.3.3
Implement detection as a small routing boundary:
- Inspect file headers first to verify the actual content, using signatures such as %PDF- and the DOCX package marker. section 5.3.3
- Fall back to the lowercased filename extension when the format lacks a reliable header signature. section 5.3.3
- Route supported formats through a parser registry, then return a common ExtractedDocument containing sections and metadata. section 5.3.3
- Use plain text as the fallback when no registered extension matches. section 5.3.3
The detector should remain independent from extraction. Each parser implements the same parse interface, while the resulting sections preserve headings, hierarchy, and page information for structure-aware processing. Adding a format therefore requires a parser implementation and registry entry without changing chunking, embedding, or storage. section 5.3.3 section 5.3.2
Listing 5.5 Format detection and parser routing
class IngestionPipeline:
def __init__(self):
self.parsers = {
"pdf": PDFParser(),
"docx": DocxParser(),
"html": HTMLParser(),
"md": MarkdownParser(),
"txt": PlainTextParser(),
}
def detect_format(self, filename: str, file_bytes: bytes) -> str:
if file_bytes[:5] == b"%PDF-": #A
return "pdf" #A
if file_bytes[:4] == b"PK\x03\x04": #A
return "docx" #A
# ... other magic byte checks for supported formats
extension = filename.rsplit(".", 1)[-1].lower() #B
if extension in self.parsers:
return extension
return "txt" #C
def extract(self, filename: str, file_bytes: bytes) -> ExtractedDocument:
format_type = self.detect_format(filename, file_bytes)
parser = self.parsers[format_type]
return parser.parse(file_bytes, filename)Shows header inspection, extension fallback, parser registration, and extraction routing in an ingestion pipeline. section 5.3.3
