Data ingestion
Data ingestion is the process of acquiring data from one or more sources and bringing it into a system for storage or processing.
Distributed Machine Learning Patterns
Data ingestion is the process of acquiring, importing, and preparing data for downstream computation or analysis.
- How it is used
- Prepares monitored and preprocessed datasets as the first stage before distributed model training.
- How deep it goes
- Defines streaming and nonstreaming modes, then develops batching, sharding, caching, workflow skipping, and a TensorFlow implementation.
- Examples
- Uses Fashion-MNIST, TensorFlow dataset loading, persistent-volume storage, and time-based cache freshness checks.
- What you can do after
- Build it from scratch as a TensorFlow ingestion step that downloads, preprocesses, persists, and conditionally reuses training data.
Read how Distributed Machine Learning Patterns explains data ingestion →
Grokking AI Applications
Document ingestion is the process of reading source documents, extracting their content, splitting it into chunks, embedding those chunks, and storing them for retrieval.
- How it is used
- Prepares document knowledge in a vector database so a RAG system can retrieve relevant chunks for answering questions.
- How deep it goes
- Breaks ingestion into reading, chunking, embedding, and storage, without detailing embedding internals.
- Examples
- Uses a pizzeria PDF containing opening hours, delivery information, and menu prices, then splits its text into chunks.
- What you can do after
- Build a Langflow ingestion subflow that extracts PDF text, chunks it, embeds it, and stores it for retrieval.
Snowflake Data Engineering
Data ingestion is the process of importing data from source locations into a system for storage or processing.
- How it is used
- Moves cloud-hosted files into Snowflake staging and target tables for scheduled or low-delay pipeline processing.
- How deep it goes
- Builds ingestion from external stages and bulk options through notification-driven Snowpipe, then defers provider-specific syntax to documentation.
- Examples
- Uses CSV cloud files for staged loading and JSON delivery-order files for notification-driven Snowpipe ingestion.
- What you can do after
- Operate/configure bulk or continuous cloud-file ingestion using external stages, COPY, Snowpipe, and staging-table checks.
Read how Snowflake Data Engineering explains data ingestion →
Designing AI Systems
Document ingestion is the process of importing a document, extracting and chunking its content, generating representations, and storing the resulting data for retrieval.
- How it is used
- Provides a shared pipeline for turning organizational documents into searchable knowledge for retrieval-augmented generation.
- How deep it goes
- Explains pipeline stages, service interfaces, re-ingestion safeguards, and asynchronous job tracking, while deferring production ownership enforcement.
- Examples
- Uses policies in PDFs, Markdown product documentation, HTML support articles, and a bulk parser update across a thousand documents.
- What you can do after
- Builds a document ingestion pipeline with configurable chunking, replacement safeguards, and asynchronous job tracking.
Delivering Machine Learning Projects: From design to deployment
Data ingestion is the process of acquiring data from source systems and bringing it into a target system for processing or storage.
- How it is used
- Supports ML data pipelines by moving acquired data into processing systems and usable modeling resources.
- How deep it goes
- Explains sources, destinations, security tradeoffs, and downstream preparation without detailing ingestion code or orchestration mechanics.
- Examples
- Uses RabbitMQ, Kafka, XLSX, CSV, Parquet, SQL queries, cloud environments, SaaS applications, and on-premises stores.
- What you can do after
- Operate/configure ingestion across heterogeneous sources while aligning target infrastructure with processing needs and security constraints.
Read how Delivering Machine Learning Projects: From design to deployment explains data ingestion →
First-Party Data Activation
The process of collecting and transferring data from source systems into a storage or processing environment.
- How it is used
- Moves organizational data into a marketing data lakehouse for customer profile management and activation.
- How deep it goes
- Distinguishes event and batch ingestion at an architecture level, without detailing implementation workflows.
- Examples
- Uses GA4, Adobe Analytics, Twilio Segment, AWS S3, Google Cloud Storage, Fivetran, and Matillion as concrete examples.
- What you can do after
- Evaluate ingestion choices by source type, delivery pattern, data identity, and downstream activation needs.
Read how First-Party Data Activation explains data ingestion →
Financial AI in Practice
Data ingestion is the process of collecting and importing data from source systems into a system for storage or processing.
- How it is used
- Connects diverse financial data sources to governed, model-ready assets for credit decisions and fraud detection.
- How deep it goes
- Explains batch and streaming patterns, safeguards, and downstream data marts, but stops short of a full production pipeline.
- Examples
- Uses banking feeds, approval events, CRM snapshots, credit data, and streamed fraud events as examples.
- What you can do after
- Understand how to choose ingestion cadence and organize validated data for downstream financial models.
Digital Twins in Action
Data ingestion is the process of collecting and importing data from one or more sources into a computing environment.
- How it is used
- Connects heterogeneous physical and enterprise data sources to a digital twin for modeling and analysis.
- How deep it goes
- Explains import tradeoffs and sliding-window processing, with a streaming example and operational considerations for industrial scale.
- Examples
- Uses a temperature sensor’s 30-second average and a public train-movement stream with a five-minute lateness window.
- What you can do after
- Evaluate whether to import data or access it in place, and apply sliding-window aggregates to a stream.
Azure Security
Data ingestion is the process of gathering and importing data from one or more sources into a system for storage and processing.
- How it is used
- Uses security data collection as the foundation for threat detection, investigation, and response in Microsoft Sentinel.
- How deep it goes
- Explains ingestion sources, workspace storage, custom APIs, and architecture decisions but stops short of deployment commands or connector configuration.
- Examples
- Uses application, operating-system, container, compute, Azure platform, subscription, tenant, and custom API log sources.
- What you can do after
- Evaluate ingestion coverage and workspace architecture for layered Azure security monitoring.
Also covered in
- API Design Patterns
- Effective Conversational AI
- Enterprise RAGIn early access (MEAP)
- Build AI-Enhanced Web Apps
- Architecting an Apache Iceberg Lakehouse
- Data Pipelines with Apache Airflow, Second Edition
- Architecture Modernization: Socio-technical alignment of software, strategy, and structure
- The Well-Grounded Data Analyst
- Elasticsearch in Action, Second Edition
- Cloud Observability in Action
- Knowledge Graphs and LLMs in Action
- DuckDB in Action
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- metadataMetadata is information that describes, identifies, or provides context about other data, resources, or artifacts.
- data sourceA data source is an origin, system, repository, or process from which data is obtained or made available for use or processing.
- similarity searchSimilarity search is a retrieval technique that finds or ranks items by their similarity to a query, often by comparing feature or vector representations under a similarity or distance measure.
Related concepts
- retrieval-augmented generationpart of · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- Snowpipeimplemented by · Snowpipe is Snowflake's service for continuously ingesting newly available files from cloud storage.
- incremental data ingestionincludes · Incremental data ingestion loads only newly created or changed records from a source system.
- pipelinepart of · A pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.
Questions about data ingestion
What is data ingestion?
Data ingestion is the process of acquiring data from one or more sources and bringing it into a system for storage or processing.
Which Manning books explain data ingestion?
Distributed Machine Learning Patterns, Grokking AI Applications, Snowflake Data Engineering, Designing AI Systems, Delivering Machine Learning Projects: From design to deployment, First-Party Data Activation, Financial AI in Practice, Digital Twins in Action and 1 more.
What should I know before learning data ingestion?
Manning books that teach data ingestion usually introduce language model, metadata, data source and similarity search first.
What kinds of data ingestion are there?
Kinds of data ingestion include incremental data ingestion, initial data ingestion and serverless data ingestion.
What is data ingestion part of?
Data ingestion is part of retrieval-augmented generation. Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
What is data ingestion also called?
Data ingestion is also called data loading.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.








