Data infrastructure
Data infrastructure is the collection of systems, tools, storage, networks, and processes for acquiring, managing, processing, and providing access to data.
Delivering Machine Learning Projects: From design to deployment
Data infrastructure is the hardware, software, storage, networking, and operational environment that supports data access and processing.
- How it is used
- Supplies accessible, transformable data for exploratory analysis and modeling, then supports models within their production data environment.
- How deep it goes
- Develops pipeline work into a ticket covering integration and tests, and explains workflow sequencing without detailing a particular implementation.
- Examples
- Illustrates integration with a two-system load, join, and null-removal DAG and a development pipeline ticket.
- What you can do after
- Build it from scratch: establish development pipelines and data tests, then replicate or reuse flows for test and production.
Effective Data Analysis
Data infrastructure is the collection of systems, tools, and processes that gathers, stores, processes, retrieves, and presents organizational data for analysis.
- How it is used
- Supports organizational analysis by moving data from varied sources into accessible, structured resources for analytics and decision-making.
- How deep it goes
- Explains collection, storage, processing, retrieval, ETL, and ELT at overview level, while deferring detailed tool implementation to documentation.
- Examples
- Uses transformed source data, renewable-energy customer data, technology comparisons, and a data-lake recommendation exercise.
- What you can do after
- Evaluate and recommend storage, collection, processing, and data-lake options for an organization’s anticipated analytical needs.
Read how Effective Data Analysis explains data infrastructure →
Related concepts
- pipelineused by · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
- platformuses · A platform is a foundational computing system, environment, or online service that provides capabilities or infrastructure for building, running, delivering, or interacting with applications and services.
- data streamproduces · A data stream is an ordered sequence of data items or events made available incrementally for consumption or processing.
- data collectionhas part · Data collection is the process of systematically acquiring or gathering information, observations, or measurements from one or more sources for a defined purpose.
- data analysisused by · Data analysis is the systematic examination, transformation, and interpretation of data to derive information, identify patterns, or support conclusions and decisions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.