Data ingestion

As explained in Delivering Machine Learning Projects: From design to deployment

Data ingestion is the process of acquiring data from source systems and bringing it into a target system for processing or storage.

Data ingestion acquires data from streams, files, databases, and separate infrastructures, then brings it into a target system for ML processing or storage. section 5.4

Plan ingestion around these delivery decisions:

• Select sources such as RabbitMQ, Kafka, XLSX, CSV, Parquet, SQL tables, cloud services, SaaS applications, or on-premises stores. section 5.4
• Choose the target infrastructure and data engines, balancing convenient cloud processing against security requirements that may keep sensitive data on premises. section 5.4
• Preserve usable downstream forms by creating intermediate tables or data stores before cleaning data and engineering features. section 5.4

Figure 5.1 The processing requirements delivered using the dev environment data pipeline infrastructure. This provides the data resources for the ML modeling team in sprint 2 and beyond.

Use the pipeline overview to connect ingestion with the processing resources delivered for ML modeling. section 5.4

How Delivering Machine Learning Projects: From design to deployment covers it

How deep it goes

Explains sources, destinations, security tradeoffs, and downstream preparation without detailing ingestion code or orchestration mechanics.

Examples

Uses RabbitMQ, Kafka, XLSX, CSV, Parquet, SQL queries, cloud environments, SaaS applications, and on-premises stores.

What you can do after

Operate/configure ingestion across heterogeneous sources while aligning target infrastructure with processing needs and security constraints.

How it connects

Connects data ingestion with ETL, data pipelines, data integration, data cleaning, feature engineering, and machine learning.

liveBookmanning.com

Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Distributed Machine Learning PatternsData ingestion is the process of acquiring, importing, and preparing data for downstream computation or analysis.
• 

  liveBookmanning.com
  Snowflake Data EngineeringData ingestion is the process of importing data from source locations into a system for storage or processing.
• 

  liveBookmanning.com
  First-Party Data ActivationThe process of collecting and transferring data from source systems into a storage or processing environment.
• 

  liveBookmanning.com
  Financial AI in PracticeData ingestion is the process of collecting and importing data from source systems into a system for storage or processing.
• 

  liveBookmanning.com
  Azure SecurityData ingestion is the process of gathering and importing data from one or more sources into a system for storage and processing.

Compare how 6 books explain data ingestion →

Related concepts

• retrieval-augmented generationpart of · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
• Snowpipeimplemented by · Snowpipe is Snowflake's service for continuously ingesting newly available files from cloud storage.
• pipelinepart of · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
• CSVuses · CSV is a plain-text format for tabular data, representing records as rows and fields as values separated by delimiters, conventionally commas, with quoting conventions for special content.
• batch pipelineincludes · A batch pipeline is a data-processing workflow that processes accumulated data in discrete runs, typically scheduled or initiated on demand, rather than continuously as it arrives.
• custom methodkind of · A custom method is a user-defined operation or capability added to an interface to provide behavior beyond its predefined functionality.

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.