liveBook · Manning

Parsing

As explained in Designing AI Systems

Parsing is the analysis of structured or semi-structured input to identify and extract its meaningful components.

Parsing turns files into clean text while preserving useful structure. This matters because PDFs can scramble columns or tables, Word files can carry revision history, and HTML can mix article content with navigation and boilerplate. A shared ingestion pipeline lets teams reuse fixes instead of maintaining separate extractors. section 5.3.1

A pipeline detects a file’s format, routes it to a matching parser, then passes extracted text onward for metadata capture and chunking. Parsers return sections with content, headings, and hierarchy; callers can derive one text string when needed. Detection checks file headers before falling back to extensions, with plain text as the fallback. section 5.3.2 section 5.3.3

Choose a parser to match the source format and its structure:

  • PDF parsers may need to handle multi-column layouts, tables, and repeated headers. section 5.3.3
  • Markdown parsers can use explicit headings, paragraph breaks, and code blocks; HTML parsers need to remove navigation and other boilerplate. section 5.3.3
  • Plain-text parsers can return a single section without a heading. section 5.3.3
Listing 5.4 Document parser interface and extracted document model
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List, Optional, Dict
 
@dataclass
class DocumentSection:
    content: str
    heading: Optional[str] = None    #A
    level: int = 0    #A
    page_number: Optional[int] = None
 
@dataclass
class ExtractedDocument:
    sections: List[DocumentSection]    #B
    metadata: Dict[str, str]    #C
 
    @property
    def text(self) -> str:    #D
        return "\n\n".join(s.content for s in self.sections)
 
class DocumentParser(ABC):
    @abstractmethod
    def parse(self, file_bytes: bytes,
              filename: str) -> ExtractedDocument:    #E
        """Extract text and structure from a document."""
        pass

The parser interface and extracted-document model show how format-specific implementations return shared structural data. section 5.3.3

Keep the pipeline extensible: implement a parser against the shared interface and register it for a new format. The rest of the pipeline can continue with chunking, embedding, and storage unchanged. section 5.3.3

How Designing AI Systems covers it

How deep it goes
Defines a parser interface, structured output, and format routing, while describing rather than implementing format-specific extraction internals.
Examples
Contrasts PDF layout issues, Markdown structure, HTML boilerplate removal, and plain text represented as one section.
What you can do after
Build format support by implementing and registering a parser that returns the shared extracted-document structure.
How it connects
Connects document parsing to format detection, metadata extraction, chunking, embedding, and vector storage.

Also explained in

Compare how 17 books explain parsing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.