What you will learn in Learn AI Data Engineering
Chapter by chapter
Chapter 1: Before You Begin
The chapter introduction frames modern AI as a major technological shift and positions it as a way for skilled professionals to reduce routine work rather than replace human judgment.
What you learn in chapter 1 →Chapter 2: Advantages & Disadvantages of Using a Coding Companion
Introduces the productivity benefits of LLM coding companions in data engineering while emphasizing that plausible-looking output requires human validation.
What you learn in chapter 2 →Chapter 3: Using a Coding Companion with SQL
The chapter introduces using ChatGPT as a coding companion for writing and debugging SQL, with prompting strategies that become more guided as queries grow more complex.
What you learn in chapter 3 →Chapter 4: Using a Coding Companion with Python
The chapter introduces using an AI coding companion to build and refine Python solutions for repetitive data engineering tasks while keeping the engineer in control.
What you learn in chapter 4 →Chapter 5: Using the OpenAI API in Data Workflows
The chapter shifts AI from an external coding companion to a component inside a data pipeline, using article sentiment analysis as an example of enrichment.
What you learn in chapter 5 →Chapter 6: AI & Data Quality
The chapter introduces AI as a way to express data quality expectations in prompts rather than stitching together separate rules and tools, then previews a reusable Python approach to validation and cleaning.
What you learn in chapter 6 →Chapter 7: AI and Advanced Data Transformations
The chapter introduces advanced data transformations as a real-world data engineering need and presents AI as a conversational way to simplify varied transformation work.
What you learn in chapter 7 →Chapter 8: AI and The Data Lifecycle
The chapter moves from using AI for individual data tasks to integrating it across a News API extract-transform-load pipeline, and introduces orchestration as a way to coordinate that pipeline.
What you learn in chapter 8 →Chapter 9: Data Cleaning and Transformation Pipelines in Practice
The chapter introduces a capstone that brings earlier AI-assisted ETL work into a coordinated pipeline for continuously arriving event data.
What you learn in chapter 9 →Chapter 10: Introduction to Web Scraping
Introduces web scraping through a product-catalog project and explains why manual extraction comes before AI-powered enrichment.
What you learn in chapter 10 →Chapter 11: AI-powered web scraping
This introduction explains why the manual RuckZone scraping approach does not scale and previews an incremental, stage-by-stage use of AI in the enrichment pipeline.
What you learn in chapter 11 →Chapter 12: AI-powered web scraping data pipelines in action
The chapter turns single-product web-scraping snippets into a batch pipeline built from single-purpose agents, with the supporting machinery needed to run it unattended.
What you learn in chapter 12 →Concepts explained in this book
- data frameA pandas DataFrame is a two-dimensional labeled tabular data structure in the pandas library.
- output formatThe response_format parameter is an API request parameter that specifies the desired structure or format of a model response.
- schemaA database schema is the formal organization of a database's tables, fields, relationships, constraints, and related definitions.
- PydanticA Pydantic model is a structured data class defined with Pydantic fields and type annotations that can validate and serialize values.
- regular expressionA regular expression is a pattern language for matching, extracting, or replacing text according to specified character and sequence rules.
- artificial intelligenceArtificial intelligence is the field of creating computational systems that perform tasks associated with human intelligence, such as reasoning, learning, perception, and decision-making.
- data transformationData transformation is the process of changing data's structure, representation, or values to meet downstream requirements.
- structured extractionAI-powered extraction is the use of an artificial intelligence model to identify and retrieve requested fields from source content based on field descriptions rather than fixed location rules.
- sentiment analysisSentiment analysis is the computational determination of opinions, emotions, or evaluative polarity expressed in language.
- JSONJSON is a text-based data interchange format that represents structured values using objects, arrays, strings, numbers, booleans, and null.
- language modelA large language model is a machine-learning model trained on large text corpora to understand and generate language and related sequences such as code.
- chat completionThe Chat Completions endpoint is an OpenAI API endpoint that generates assistant responses from conversational messages and related request parameters.
- data contractA data contract is a formal specification of the structure, fields, types, and validity requirements that data exchanged between processing components must satisfy.
- scrapingWeb scraping is the automated or manual extraction of information from web pages, typically by processing their underlying markup.
- prompt engineeringPrompt engineering is the deliberate design and refinement of inputs provided to an AI model to influence the quality, format, or behavior of its outputs.
- SQLA search query is a set of terms or structured input submitted to a search system to retrieve relevant results.
- requestsRequests is a Python library for sending HTTP requests and handling their responses.
- hallucinationAn AI hallucination is a confident but incorrect or misleading output produced by an artificial intelligence model.
- zero-shot promptingZero-shot prompting is a prompting technique that asks a model to perform a task without providing task-specific examples.
- Beautiful SoupBeautifulSoup is a Python library for parsing HTML and XML documents into navigable structures and extracting or modifying their contents.
- entity resolutionEntity resolution is the process of determining which records refer to the same real-world entity despite differences or errors in their attributes.
- dataclassA data class is a class whose primary purpose is representing structured data, typically with declarative fields and automatically supplied data-oriented behavior.
- NumbersAPINumbersAPI is a public web API that returns trivia, mathematical, date, and year-related facts about numbers.
- pitch typeA pitch type is a baseball classification describing how a particular pitch is thrown and moves, such as a curveball.
- data enrichmentEnriched data is information added to source data through inference, generation, or incorporation from external sources.
- excitement scoreAn excitement score is a numerical rating representing the perceived dramatic intensity of a baseball play on a specified scale.
- PythonPython is a general-purpose programming language widely used for automation, data processing, and artificial-intelligence applications.
- HTMLHTML is a markup language for structuring content in web documents, including the source text retrieved from a web server.
- url discoveryURL discovery is the process of finding and selecting a web address for a particular product or other target item.
- HTML cleaningHTML cleaning is the removal of irrelevant, distracting, or unsafe markup and content from an HTML document before further processing.
- NewsAPINewsAPI is a public HTTP service that searches and retrieves articles from news sources and blogs.
- product extractionProduct extraction is the process of extracting product attributes from source content.
- computational notebookJupyter is an open-source interactive computing project providing notebook-based tools for writing and executing code with explanatory content and outputs.
- site-specific selectorA site-specific selector is a webpage selection rule tailored to the HTML structure of a particular website.
- chain-of-thought promptingChain-of-thought prompting is a prompting technique that asks a language model to produce or follow an explicit sequence of intermediate reasoning steps before an answer.
- natural language interfaceA natural language interface allows users to interact with software through ordinary human language rather than a specialized command syntax.
- review queueA review queue is an ordered collection of items awaiting inspection or resolution by a reviewer.
- data cleaningData cleaning is the process of detecting and correcting, removing, or standardizing inaccurate, incomplete, duplicated, or unsuitable data.
- Session classA Requests session is an object that persists settings and can reuse connections across multiple HTTP requests.
- payment termsNet payment terms specify the number of days allowed between an invoice and its required payment date.
- percentage contributionA percentage contribution is the proportion of a specified total attributable to an item, account, or component, expressed as a percentage.
- retryRetry logic is a control technique that repeats a failed operation according to defined attempt and delay policies.
- context windowA context window is the finite amount of text or tokens that a language model can process in one interaction.
- unstructured dataUnstructured data is information that lacks a consistently organized schema for its content and fields.
- normalized tableA normalized table is a flat tabular representation in which nested records are expanded into rows with their relevant fields as columns.
- LogExtraction modelThe LogExtraction model is a structured data model representing extracted log type, date, time, and message fields.
- manual extractionManual extraction is the creation and maintenance of explicit rules for retrieving selected fields from source pages.
- AI-assisted ranking and filteringAI ranking is the evaluation and ordering of candidate items by an artificial intelligence model according to specified criteria.
- edge caseAn edge case is an unusual, extreme, or boundary condition that can expose limitations or failures in a system or process.
- self-consistencySelf-consistency prompting is a prompting technique that elicits multiple solution paths and compares or aggregates them to improve answer reliability.
- pitch velocityThe mean speed of a pitcher's pitches measured in miles per hour.
- PipelineRecordPipelineRecord is a structured model that wraps a product with bookkeeping fields for tracking pipeline progress, confidence, extracted data, validation, and failure state.
- phone numberA U.S. phone number is a telephone number represented according to numbering and formatting conventions used in the United States.
- missing valueA null value is a marker representing the absence or unknown status of a value in a data field.
- timezone conversionTimezone conversion is the transformation of a timestamp from one timezone representation to another while preserving the represented instant.
- status trackingStatus tracking is the recording and monitoring of the current state of tasks, records, or processing outcomes.
- visualizationData visualization is the graphical representation of data to reveal patterns, relationships, trends, or comparisons.
- fallbackA fallback strategy is an approach that specifies an alternative method when the preferred method fails or cannot produce a result.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
