liveBook · Manning

Format detection

As explained in Designing AI Systems

Format detection is the process of determining the file format of an input document before selecting a parser.

Format detection identifies an input file before parsing so an ingestion pipeline can route it to the parser responsible for that format. The parser then produces clean text while preserving structure for downstream chunking. section 5.3.2 section 5.3.3

Implement detection as a small routing boundary:

  • Inspect file headers first to verify the actual content, using signatures such as %PDF- and the DOCX package marker. section 5.3.3
  • Fall back to the lowercased filename extension when the format lacks a reliable header signature. section 5.3.3
  • Route supported formats through a parser registry, then return a common ExtractedDocument containing sections and metadata. section 5.3.3
  • Use plain text as the fallback when no registered extension matches. section 5.3.3

The detector should remain independent from extraction. Each parser implements the same parse interface, while the resulting sections preserve headings, hierarchy, and page information for structure-aware processing. Adding a format therefore requires a parser implementation and registry entry without changing chunking, embedding, or storage. section 5.3.3 section 5.3.2

Listing 5.5 Format detection and parser routing
class IngestionPipeline:
    def __init__(self):
        self.parsers = {
            "pdf": PDFParser(),
            "docx": DocxParser(),
            "html": HTMLParser(),
            "md": MarkdownParser(),
            "txt": PlainTextParser(),
        }
 
    def detect_format(self, filename: str, file_bytes: bytes) -> str:
        if file_bytes[:5] == b"%PDF-":    #A
            return "pdf"    #A
        if file_bytes[:4] == b"PK\x03\x04":    #A
            return "docx"    #A
        # ... other magic byte checks for supported formats
 
        extension = filename.rsplit(".", 1)[-1].lower()    #B
        if extension in self.parsers:
            return extension
 
        return "txt"    #C
 
    def extract(self, filename: str, file_bytes: bytes) -> ExtractedDocument:
        format_type = self.detect_format(filename, file_bytes)
        parser = self.parsers[format_type]
        return parser.parse(file_bytes, filename)

Shows header inspection, extension fallback, parser registration, and extraction routing in an ingestion pipeline. section 5.3.3

How Designing AI Systems covers it

How deep it goes
Explains header checks, extension fallback, parser registries, and structural outputs, but defers format-specific parsing internals.
Examples
Uses PDF, DOCX, HTML, Markdown, and plain-text inputs, including an insurance-policy PDF with headed sections.
What it assumes
Assumes familiarity with ingestion pipelines and parsers while defining the shared extraction model and routing boundary.
What you can do after
Build it from scratch by implementing detection, registering parsers, and routing files into a shared extracted-document model.
How it connects
Connects format detection with text extraction, document parsers, structural chunking, embeddings, and vector storage.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.