Data ingestion
Data ingestion is the process of acquiring or receiving data from one or more sources and bringing it into a system for storage or processing.
Distributed Machine Learning Patterns
Data ingestion is the process of acquiring, importing, and preparing data for downstream computation or analysis.
- How it is used
- Prepares monitored and preprocessed datasets as the first stage before distributed model training.
- How deep it goes
- Defines streaming and nonstreaming modes, then develops batching, sharding, caching, workflow skipping, and a TensorFlow implementation.
- Examples
- Uses Fashion-MNIST, TensorFlow dataset loading, persistent-volume storage, and time-based cache freshness checks.
- What you can do after
- Build it from scratch as a TensorFlow ingestion step that downloads, preprocesses, persists, and conditionally reuses training data.
Read how Distributed Machine Learning Patterns explains data ingestion →
Snowflake Data Engineering
Data ingestion is the process of importing data from source locations into a system for storage or processing.
- How it is used
- Moves cloud-hosted files into Snowflake staging and target tables for scheduled or low-delay pipeline processing.
- How deep it goes
- Builds ingestion from external stages and bulk options through notification-driven Snowpipe, then defers provider-specific syntax to documentation.
- Examples
- Uses CSV cloud files for staged loading and JSON delivery-order files for notification-driven Snowpipe ingestion.
- What you can do after
- Operate/configure bulk or continuous cloud-file ingestion using external stages, COPY, Snowpipe, and staging-table checks.
Read how Snowflake Data Engineering explains data ingestion →
Delivering Machine Learning Projects: From design to deployment
Data ingestion is the process of acquiring data from source systems and bringing it into a target system for processing or storage.
- How it is used
- Supports ML data pipelines by moving acquired data into processing systems and usable modeling resources.
- How deep it goes
- Explains sources, destinations, security tradeoffs, and downstream preparation without detailing ingestion code or orchestration mechanics.
- Examples
- Uses RabbitMQ, Kafka, XLSX, CSV, Parquet, SQL queries, cloud environments, SaaS applications, and on-premises stores.
- What you can do after
- Operate/configure ingestion across heterogeneous sources while aligning target infrastructure with processing needs and security constraints.
Read how Delivering Machine Learning Projects: From design to deployment explains data ingestion →
First-Party Data Activation
The process of collecting and transferring data from source systems into a storage or processing environment.
- How it is used
- Moves organizational data into a marketing data lakehouse for customer profile management and activation.
- How deep it goes
- Distinguishes event and batch ingestion at an architecture level, without detailing implementation workflows.
- Examples
- Uses GA4, Adobe Analytics, Twilio Segment, AWS S3, Google Cloud Storage, Fivetran, and Matillion as concrete examples.
- What you can do after
- Evaluate ingestion choices by source type, delivery pattern, data identity, and downstream activation needs.
Read how First-Party Data Activation explains data ingestion →
Financial AI in Practice
Data ingestion is the process of collecting and importing data from source systems into a system for storage or processing.
- How it is used
- Connects diverse financial data sources to governed, model-ready assets for credit decisions and fraud detection.
- How deep it goes
- Explains batch and streaming patterns, safeguards, and downstream data marts, but stops short of a full production pipeline.
- Examples
- Uses banking feeds, approval events, CRM snapshots, credit data, and streamed fraud events as examples.
- What you can do after
- Understand how to choose ingestion cadence and organize validated data for downstream financial models.
Azure Security
Data ingestion is the process of gathering and importing data from one or more sources into a system for storage and processing.
- How it is used
- Uses security data collection as the foundation for threat detection, investigation, and response in Microsoft Sentinel.
- How deep it goes
- Explains ingestion sources, workspace storage, custom APIs, and architecture decisions but stops short of deployment commands or connector configuration.
- Examples
- Uses application, operating-system, container, compute, Azure platform, subscription, tenant, and custom API log sources.
- What you can do after
- Evaluate ingestion coverage and workspace architecture for layered Azure security monitoring.
Related concepts
- retrieval-augmented generationpart of · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- Snowpipeimplemented by · Snowpipe is Snowflake's service for continuously ingesting newly available files from cloud storage.
- pipelinepart of · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
- CSVuses · CSV is a plain-text format for tabular data, representing records as rows and fields as values separated by delimiters, conventionally commas, with quoting conventions for special content.
- batch pipelineincludes · A batch pipeline is a data-processing workflow that processes accumulated data in discrete runs, typically scheduled or initiated on demand, rather than continuously as it arrives.
- custom methodkind of · A custom method is a user-defined operation or capability added to an interface to provide behavior beyond its predefined functionality.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.