A large language model (LLM) generates and reasons over natural language through machine-learned patterns. In data engineering, it can serve first as a coding companion for SQL, Python, debugging, and iterative refinement, then as a pipeline component that classifies, summarizes, cleans, reshapes, or extracts data from natural-language inputs. section 1.2.2 section 1.1 section 5.0
Treat an LLM as an assistant, not an authority. It can produce plausible but incorrect code, fields, joins, or business logic because it does not truly understand the actual schema, relationships, architecture, or goals. Validate output against schemas and business context, use feedback loops to refine prompts, and keep generated results out of production until checked. section 2.0 section 2.2
LLMs fit best where judgment over variable, unstructured input is worth the cost and review effort:
- Use them to accelerate complex code generation, repetitive scripting, pipeline scaffolding, and rapid prototyping. section 1.1 section 2.2
- Embed them as enrichment steps for text classification, such as sentiment labels that can flow into Pandas, scikit-learn, PostgreSQL, or Tableau. section 5.0
- Use schema-guided extraction on cleaned HTML when many source layouts would otherwise require custom selectors, regex, and site-specific rules. section 11.6.2
- Prefer deterministic code for simple transformations, exact matching, joins, or high-volume low-value decisions; account for per-call cost, latency, and fallback handling. section 11.9 section 11.6.2
Extracting product fields from variable web pages
Send cleaned product-page HTML together with a Pydantic schema and explicit rules: extract only text that is present, return null when a field is unavailable or uncertain, and do not guess. The model can return product name, brand, description, price, weight, image URL, and category without source-specific selectors. Schema checks, spot samples, known-value comparisons, confidence scoring, status tracking, review queues, and fallback logic make the step operationally safer. section 11.6.2 section 11.9
Figure 11.6 Flowchart: "Should I use AI for this step?" Decision tree walking through the key questions from the checklist.
Use the decision tree to evaluate whether an AI step has unstructured input, variable source logic, justified cost, and a practical validation path. section 11.9
























































































