liveBook · Manning

Language model

As explained in Learn AI Data Engineering

A large language model is a machine-learning model trained on large text corpora to understand and generate language and related sequences such as code.

A large language model (LLM) generates and reasons over natural language through machine-learned patterns. In data engineering, it can serve first as a coding companion for SQL, Python, debugging, and iterative refinement, then as a pipeline component that classifies, summarizes, cleans, reshapes, or extracts data from natural-language inputs. section 1.2.2 section 1.1 section 5.0

Treat an LLM as an assistant, not an authority. It can produce plausible but incorrect code, fields, joins, or business logic because it does not truly understand the actual schema, relationships, architecture, or goals. Validate output against schemas and business context, use feedback loops to refine prompts, and keep generated results out of production until checked. section 2.0 section 2.2

LLMs fit best where judgment over variable, unstructured input is worth the cost and review effort:

  • Use them to accelerate complex code generation, repetitive scripting, pipeline scaffolding, and rapid prototyping. section 1.1 section 2.2
  • Embed them as enrichment steps for text classification, such as sentiment labels that can flow into Pandas, scikit-learn, PostgreSQL, or Tableau. section 5.0
  • Use schema-guided extraction on cleaned HTML when many source layouts would otherwise require custom selectors, regex, and site-specific rules. section 11.6.2
  • Prefer deterministic code for simple transformations, exact matching, joins, or high-volume low-value decisions; account for per-call cost, latency, and fallback handling. section 11.9 section 11.6.2

Extracting product fields from variable web pages

Send cleaned product-page HTML together with a Pydantic schema and explicit rules: extract only text that is present, return null when a field is unavailable or uncertain, and do not guess. The model can return product name, brand, description, price, weight, image URL, and category without source-specific selectors. Schema checks, spot samples, known-value comparisons, confidence scoring, status tracking, review queues, and fallback logic make the step operationally safer. section 11.6.2 section 11.9

Figure 11.6 Flowchart: "Should I use AI for this step?" Decision tree walking through the key questions from the checklist.

Use the decision tree to evaluate whether an AI step has unstructured input, variable source logic, justified cost, and a practical validation path. section 11.9

How Learn AI Data Engineering covers it

How deep it goes
Explains practical prompting, schema-guided extraction, validation, cost, and fallback decisions rather than model internals.
Examples
Demonstrates sentiment-label enrichment for news articles and schema-constrained product extraction from cleaned HTML.
What it assumes
Assumes familiarity with data pipelines and coding tasks while framing LLM output as reviewable first drafts.
What you can do after
Apply it via an API as a validated pipeline step for classification or structured extraction.
How it connects
Connects LLMs to SQL, Python, Pandas, Pydantic, sentiment analysis, web scraping, and hallucinations.

Also explained in

Compare how 89 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.