Unstructured data lacks a consistent schema for its content and fields. It includes free text, HTML, PDFs, images, and noisy event records whose useful information must be identified before structured processing. section 11.2 section 9.0
Treat unstructured data as a pipeline-design problem:
- Use deterministic code for simple transformations, exact matching, joins, and other operations with stable rules. section 11.9
- Use AI when content varies by source, requires human-like judgment, or must be mapped into a defined schema. section 11.2 section 11.9
- Validate extracted results with schema checks, spot samples, known-value comparisons, confidence scores, status tracking, review queues, and fallback logic. section 11.9
- Compare per-record cost and latency against rules or regex before sending high-volume data to an LLM. section 11.9 section 11.3
Product-page extraction
A product page contains an unstructured title, price, description, and images. A schema-driven prompt can extract those fields, while deterministic validation checks types and known values; review queues handle uncertain results. section 11.2 section 11.3
Figure 11.6 Flowchart: "Should I use AI for this step?" Decision tree walking through the key questions from the checklist.
Use the decision tree to determine whether an unstructured-data step benefits from AI or should remain deterministic. section 11.9












