Chapter 1: Before You Begin
The chapter introduction frames modern AI as a major technological shift and positions it as a way for skilled professionals to reduce routine work rather than replace human judgment.
After this chapter you will be able to
- AI as professional augmentation. Describe the book's stance that AI should enhance skilled professionals rather than replace human intelligence: it can automate mundane tasks so people can focus on creative, critical, and problem-solving work. Distinguish this focus from the unresolved question of whether prompt engineering could replace development teams.
- Shift toward business value. Explain how AI handling repetitive or infrastructure-related work may let data engineers spend more time on business logic, insight, and impact.
- Coding companion and reviewer. Describe AI as a coding companion that can generate repetitive scripts or pipeline scaffolding and also help review prompts, debug code, and compare implementation options. Explain why the section calls it a multi-tool rather than a shortcut that eliminates human intervention.
- Programmatic AI for Data Workflows. Explain that the book covers AI coding companions but places greater emphasis on programmatic AI for data ingestion, transformation, and enrichment at scale.
- AI roles in data engineering. Explain how AI can help data engineers clean and transform data, extract information from unstructured data, generate synthetic test data, perform language analysis, and support data governance.
- Shared Function of LLMs. Explain that LLMs differ in capabilities, design philosophies, and openness, but share a core function of generating and reasoning over natural language using machine-learned patterns.
- Daily reading and practice rhythm. Describe the Month of Lunches format: complete one chapter per day, spending about 40 minutes reading and 20 minutes practicing, without needing to read at lunchtime.
- Main chapter progression. Describe how the main chapters progress from AI coding companions and prompt engineering, through data transformations, feature extraction, and AI-powered automation, to structured data extraction, agentic workflows, and programmatic AI applications.
- Chapter setup guide use. Describe where to find the chapter setup guides, what preparation they cover, and why to follow the relevant guide before starting a hands-on lab.
- Environment setup requirements. State that following the examples requires PostgreSQL and pgAdmin for SQL exercises, Jupyter Lab for Python workflows, and an OpenAI account for AI-powered tasks. Describe the linked GitHub repository as the source of setup instructions.
- OpenAI API key setup. Describe signing up with OpenAI and generating an API token to follow the AI-driven examples, and identify the linked setup guide as a source of detailed instructions.
- Practical AI data engineering focus. Explain that the book prioritizes immediate, hands-on use of AI over deep study of LLM architectures or theory, and describe the data engineering work readers will practice.
Section by section
Introduction
The chapter introduction frames modern AI as a major technological shift and positions it as a way for skilled professionals to reduce routine work rather than replace human judgment.
- Modern AI's technological shift: Explain why the introduction presents modern AI as a more consequential shift than earlier AI waves: earlier efforts were constrained by limited computing power and rigid rules, while today's systems benefit from scalable models, vast datasets, and practical applications.
Why it matters: This context explains why AI is changing professional work now. - AI as professional augmentation: Describe the book's stance that AI should enhance skilled professionals rather than replace human intelligence: it can automate mundane tasks so people can focus on creative, critical, and problem-solving work. Distinguish this focus from the unresolved question of whether prompt engineering could replace development teams.
Why it matters: This stance sets the purpose for using AI throughout the book.
Key terms: limited compute power, rule-based systems, scalable models, vast datasets, real-world applications, enhances rather than replaces human intelligence, mundane tasks, high-value work, prompt engineering, agentic systems
1.1 Why AI Matters to Data Engineering
This section explains why AI matters to data engineering: it can take on repetitive work, support coding and review, and help engineers focus on business value.
- Shift toward business value: Explain how AI handling repetitive or infrastructure-related work may let data engineers spend more time on business logic, insight, and impact.
Why it matters: This shift explains why AI changes the focus of data engineering work, not just its speed. - Coding companion and reviewer: Describe AI as a coding companion that can generate repetitive scripts or pipeline scaffolding and also help review prompts, debug code, and compare implementation options. Explain why the section calls it a multi-tool rather than a shortcut that eliminates human intervention.
Why it matters: AI’s value here includes both producing work and helping engineers assess it. - AI across data roles: Describe how data engineers support analysts and data scientists, and give the section’s examples of AI helping each role: SQL for analysts, feature suggestions for scientists, and raw-data transformation for engineers.
Why it matters: The comparison shows that AI’s usefulness extends across the data lifecycle.
Key terms: data engineering, business transformations, business value, infrastructure tasks, coding companion, large language models, scaffolding, multi-tool, human intervention, data personas
1.2 Is This Book for You?
This section identifies the book’s intended readers and explains its focus on applying AI programmatically in data workflows rather than stopping at interactive prompts.
- Programmatic AI for Data Workflows: Explain that the book covers AI coding companions but places greater emphasis on programmatic AI for data ingestion, transformation, and enrichment at scale.
Why it matters: This distinction explains what readers will work toward beyond using AI in a chat or IDE. - Intended Readers and Starting Skills: Describe the readers this book addresses—data engineers, analysts, data scientists, and AI enthusiasts—and the work they may want to do. State that SQL, Python, and AI familiarity help readers start quickly, but the practical, hands-on approach is intended to remain accessible to a broad audience.
Why it matters: Readers can judge how their experience connects to the book’s examples without treating prior expertise as a strict requirement.
Key terms: AI coding companions, programmatic applications, data ingestion, transformation, enrichment, data engineers, analysts, data scientists, AI enthusiasts, SQL
1.2.1 The Many Uses for AI
This section surveys familiar uses of AI and introduces the ways it can support data engineering work.
- Everyday and business AI applications: Describe how AI supports familiar services and business activities, using examples such as virtual assistants, recommendations, fraud detection, diagnostics, or customer-behavior prediction.
Why it matters: These examples show that AI already serves many purposes beyond writing code. - AI roles in data engineering: Explain how AI can help data engineers clean and transform data, extract information from unstructured data, generate synthetic test data, perform language analysis, and support data governance.
Why it matters: These are the data-engineering applications the book goes on to explore.
Key terms: virtual assistants, personalized recommendations, fraud detection, Google Vertex AI, data cleansing, transformation, unstructured data, synthetic datasets, entity recognition, sentiment analysis
1.2.2 The Many Flavors of AI
This section introduces LLMs as a varied ecosystem, explains their shared function, and states why the book primarily uses OpenAI’s GPT models.
- Shared Function of LLMs: Explain that LLMs differ in capabilities, design philosophies, and openness, but share a core function of generating and reasoning over natural language using machine-learned patterns.
Why it matters: This gives you a common way to understand the different AI tools used throughout the book. - GPT Focus and Model Choice: Explain why the book focuses primarily on OpenAI’s GPT models while still encouraging awareness of other LLM providers and their trade-offs when choosing a tool for a use case.
Why it matters: The book’s choice of model does not mean it is the right choice for every task.
Key terms: large language models (LLMs), natural language, machine-learned patterns, OpenAI’s GPT models, LLM providers, strengths and trade-offs
1.3 How to Use This Book
This section explains the book’s one-chapter-per-day reading and practice rhythm.
- Daily reading and practice rhythm: Describe the Month of Lunches format: complete one chapter per day, spending about 40 minutes reading and 20 minutes practicing, without needing to read at lunchtime.
Why it matters: The schedule gives you a manageable way to read and practice consistently.
Key terms: Month of Lunches, one chapter per day
1.3.1 The Main Chapters
This subsection maps the main chapters from foundational AI concepts through data engineering applications to advanced workflows.
- Main chapter progression: Describe how the main chapters progress from AI coding companions and prompt engineering, through data transformations, feature extraction, and AI-powered automation, to structured data extraction, agentic workflows, and programmatic AI applications.
Why it matters: The progression helps you see how the book moves from foundational ideas to more advanced applications.
Key terms: AI coding companions, prompt engineering, data transformations, feature extraction, AI-powered automation, structured data extraction
1.3.2 Hands-on Labs
This section explains that the book’s short labs are hands-on practice and encourages readers to try them before consulting the solutions.
1.3.3 Chapter Setup Files
This section directs readers to chapter-specific GitHub setup guides so they can prepare for hands-on labs without getting stuck on configuration.
- Chapter setup guide use: Describe where to find the chapter setup guides, what preparation they cover, and why to follow the relevant guide before starting a hands-on lab.
Why it matters: The guides help you get each chapter’s environment ready and troubleshoot it later.
Key terms: setup/ directory, companion GitHub repository, environment variables, API key management
1.4 Setting Up Your Environment
This section identifies the local tools and account needed to follow the book’s examples and points readers to setup instructions.
- Environment setup requirements: State that following the examples requires PostgreSQL and pgAdmin for SQL exercises, Jupyter Lab for Python workflows, and an OpenAI account for AI-powered tasks. Describe the linked GitHub repository as the source of setup instructions.
Why it matters: Completing this setup lets you follow along with the book’s examples.
Key terms: PostgreSQL, pgAdmin, Jupyter Lab, OpenAI account, GitHub repo
1.4.1 Installing PostgreSQL and pgAdmin
1.4.2 Installing Jupyter Lab for Python Work
1.4.3 Creating an OpenAI Account
This subsection tells readers how to obtain the OpenAI API key needed for the book’s AI-driven examples.
- OpenAI API key setup: Describe signing up with OpenAI and generating an API token to follow the AI-driven examples, and identify the linked setup guide as a source of detailed instructions.
Why it matters: The examples that use OpenAI’s API require an API key.
Key terms: OpenAI API key, API token, setup guide
1.5 Being Immediately Effective with AI and Data Engineering
This section presents the book as a hands-on guide to using AI immediately in data engineering work rather than studying AI theory in depth.
- Practical AI data engineering focus: Explain that the book prioritizes immediate, hands-on use of AI over deep study of LLM architectures or theory, and describe the data engineering work readers will practice.
Why it matters: This focus helps you connect the book’s examples to tasks you can use in your own work.
Key terms: practical application, LLM architectures, generate and validate SQL queries, clean and transform datasets, unstructured data, feature engineering
Where this is used later
- Mental Model: The Data Engineer and the Coding Companion (chapter 2) · The data engineer–coding companion mental model develops the relationship between a skilled professional and an AI tool.
- Disadvantages of Using a Coding Companion (chapter 2) · Examines disadvantages that qualify the companion’s usefulness.
- Identifying Data Quality Issues (chapter 6) · Develops the use of AI to identify data quality issues.
- Synthetic Data (chapter 11) · Develops the use of synthetic data.
- Understanding the ChatGPT API and Chat Completions Endpoint (chapter 5) · The discussion of the ChatGPT API uses the book’s chosen model family.
- Lab (chapter 2) · Readers prepare to complete a hands-on lab.
- Zero-Shot Prompting (chapter 3) · Begins the hands-on work of using AI to generate SQL.
- Evolving AI Integration (chapter 8) · Develops the promised integration of AI into data workflows.
Concepts explained in this chapter
- artificial intelligenceArtificial intelligence is the field of creating computational systems that perform tasks associated with human intelligence, such as reasoning, learning, perception, and decision-making.
- language modelA large language model is a machine-learning model trained on large text corpora to understand and generate language and related sequences such as code.
- data transformationData transformation is the process of changing data's structure, representation, or values to meet downstream requirements.
- prompt engineeringPrompt engineering is the deliberate design and refinement of inputs provided to an AI model to influence the quality, format, or behavior of its outputs.
- data cleaningData cleaning is the process of detecting and correcting, removing, or standardizing inaccurate, incomplete, duplicated, or unsuitable data.
- SQLA search query is a set of terms or structured input submitted to a search system to retrieve relevant results.
- data enrichmentEnriched data is information added to source data through inference, generation, or incorporation from external sources.
- PythonPython is a general-purpose programming language widely used for automation, data processing, and artificial-intelligence applications.
- structured extractionAI-powered extraction is the use of an artificial intelligence model to identify and retrieve requested fields from source content based on field descriptions rather than fixed location rules.
- natural language interfaceA natural language interface allows users to interact with software through ordinary human language rather than a specialized command syntax.
- sentiment analysisSentiment analysis is the computational determination of opinions, emotions, or evaluative polarity expressed in language.
- unstructured dataUnstructured data is information that lacks a consistently organized schema for its content and fields.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
