liveBook · Manning

Data ingestion

Also called data loading

Data ingestion is the process of acquiring data from one or more sources and bringing it into a system for storage or processing.

How 9 Manning books explain it

Distributed Machine Learning Patterns

Data ingestion is the process of acquiring, importing, and preparing data for downstream computation or analysis.

How it is used
Prepares monitored and preprocessed datasets as the first stage before distributed model training.
How deep it goes
Defines streaming and nonstreaming modes, then develops batching, sharding, caching, workflow skipping, and a TensorFlow implementation.
Examples
Uses Fashion-MNIST, TensorFlow dataset loading, persistent-volume storage, and time-based cache freshness checks.
What you can do after
Build it from scratch as a TensorFlow ingestion step that downloads, preprocesses, persists, and conditionally reuses training data.

Read how Distributed Machine Learning Patterns explains data ingestion →

Grokking AI Applications

Document ingestion is the process of reading source documents, extracting their content, splitting it into chunks, embedding those chunks, and storing them for retrieval.

How it is used
Prepares document knowledge in a vector database so a RAG system can retrieve relevant chunks for answering questions.
How deep it goes
Breaks ingestion into reading, chunking, embedding, and storage, without detailing embedding internals.
Examples
Uses a pizzeria PDF containing opening hours, delivery information, and menu prices, then splits its text into chunks.
What you can do after
Build a Langflow ingestion subflow that extracts PDF text, chunks it, embeds it, and stores it for retrieval.

Read how Grokking AI Applications explains data ingestion →

Snowflake Data Engineering

Data ingestion is the process of importing data from source locations into a system for storage or processing.

How it is used
Moves cloud-hosted files into Snowflake staging and target tables for scheduled or low-delay pipeline processing.
How deep it goes
Builds ingestion from external stages and bulk options through notification-driven Snowpipe, then defers provider-specific syntax to documentation.
Examples
Uses CSV cloud files for staged loading and JSON delivery-order files for notification-driven Snowpipe ingestion.
What you can do after
Operate/configure bulk or continuous cloud-file ingestion using external stages, COPY, Snowpipe, and staging-table checks.

Read how Snowflake Data Engineering explains data ingestion →

Designing AI Systems

Document ingestion is the process of importing a document, extracting and chunking its content, generating representations, and storing the resulting data for retrieval.

How it is used
Provides a shared pipeline for turning organizational documents into searchable knowledge for retrieval-augmented generation.
How deep it goes
Explains pipeline stages, service interfaces, re-ingestion safeguards, and asynchronous job tracking, while deferring production ownership enforcement.
Examples
Uses policies in PDFs, Markdown product documentation, HTML support articles, and a bulk parser update across a thousand documents.
What you can do after
Builds a document ingestion pipeline with configurable chunking, replacement safeguards, and asynchronous job tracking.

Read how Designing AI Systems explains data ingestion →

Delivering Machine Learning Projects: From design to deployment

Data ingestion is the process of acquiring data from source systems and bringing it into a target system for processing or storage.

How it is used
Supports ML data pipelines by moving acquired data into processing systems and usable modeling resources.
How deep it goes
Explains sources, destinations, security tradeoffs, and downstream preparation without detailing ingestion code or orchestration mechanics.
Examples
Uses RabbitMQ, Kafka, XLSX, CSV, Parquet, SQL queries, cloud environments, SaaS applications, and on-premises stores.
What you can do after
Operate/configure ingestion across heterogeneous sources while aligning target infrastructure with processing needs and security constraints.

Read how Delivering Machine Learning Projects: From design to deployment explains data ingestion →

First-Party Data Activation

The process of collecting and transferring data from source systems into a storage or processing environment.

How it is used
Moves organizational data into a marketing data lakehouse for customer profile management and activation.
How deep it goes
Distinguishes event and batch ingestion at an architecture level, without detailing implementation workflows.
Examples
Uses GA4, Adobe Analytics, Twilio Segment, AWS S3, Google Cloud Storage, Fivetran, and Matillion as concrete examples.
What you can do after
Evaluate ingestion choices by source type, delivery pattern, data identity, and downstream activation needs.

Read how First-Party Data Activation explains data ingestion →

Financial AI in Practice

Data ingestion is the process of collecting and importing data from source systems into a system for storage or processing.

How it is used
Connects diverse financial data sources to governed, model-ready assets for credit decisions and fraud detection.
How deep it goes
Explains batch and streaming patterns, safeguards, and downstream data marts, but stops short of a full production pipeline.
Examples
Uses banking feeds, approval events, CRM snapshots, credit data, and streamed fraud events as examples.
What you can do after
Understand how to choose ingestion cadence and organize validated data for downstream financial models.

Read how Financial AI in Practice explains data ingestion →

Digital Twins in Action

Data ingestion is the process of collecting and importing data from one or more sources into a computing environment.

How it is used
Connects heterogeneous physical and enterprise data sources to a digital twin for modeling and analysis.
How deep it goes
Explains import tradeoffs and sliding-window processing, with a streaming example and operational considerations for industrial scale.
Examples
Uses a temperature sensor’s 30-second average and a public train-movement stream with a five-minute lateness window.
What you can do after
Evaluate whether to import data or access it in place, and apply sliding-window aggregates to a stream.

Read how Digital Twins in Action explains data ingestion →

Azure Security

Data ingestion is the process of gathering and importing data from one or more sources into a system for storage and processing.

How it is used
Uses security data collection as the foundation for threat detection, investigation, and response in Microsoft Sentinel.
How deep it goes
Explains ingestion sources, workspace storage, custom APIs, and architecture decisions but stops short of deployment commands or connector configuration.
Examples
Uses application, operating-system, container, compute, Azure platform, subscription, tenant, and custom API log sources.
What you can do after
Evaluate ingestion coverage and workspace architecture for layered Azure security monitoring.

Read how Azure Security explains data ingestion →

Also covered in

What to know first

Related concepts

Questions about data ingestion

What is data ingestion?

Data ingestion is the process of acquiring data from one or more sources and bringing it into a system for storage or processing.

Which Manning books explain data ingestion?

Distributed Machine Learning Patterns, Grokking AI Applications, Snowflake Data Engineering, Designing AI Systems, Delivering Machine Learning Projects: From design to deployment, First-Party Data Activation, Financial AI in Practice, Digital Twins in Action and 1 more.

What should I know before learning data ingestion?

Manning books that teach data ingestion usually introduce language model, metadata, data source and similarity search first.

What kinds of data ingestion are there?

Kinds of data ingestion include incremental data ingestion, initial data ingestion and serverless data ingestion.

What is data ingestion part of?

Data ingestion is part of retrieval-augmented generation. Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.

What is data ingestion also called?

Data ingestion is also called data loading.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.