Data infrastructure gives an ML team reliable access to the data it needs to investigate, transform, and use. That data may be spread across databases, warehouses, containers, clouds, and organizations. Before building pipelines, obtain a representative sample and check whether it spans the relevant time periods and source systems. Even obtaining the sample can expose mistaken assumptions about where data lives and how it can be accessed. section 2.4 section 5.4
A flexible data pipeline is central to this infrastructure. It chains transformations through a workflow engine and scheduler, allowing the team to respond as data resources or modeling needs change. Unlike integration centered on a relational warehouse, pipelines can bring together varied sources, including unstructured data. The immediate goal is a development environment where exploratory data analysis is quick and reproducible; the same flow must later be replicated or reused for test and production. section 4.3.3 section 5.4
Turn that goal into an infrastructure ticket:
- Survey available data and its integrity before relying on it for modeling. section 5.6
- Aggregate and transform relevant assets into usable records or streams; standardize or filter unstructured items where needed. section 5.4
- Provide the modeling team access to the results, and establish data and pipeline tests so later work uses the intended data. section 5.6
A small integration workflow
One workflow loads tables from two systems, joins them, then removes null values. Each step runs a script on a target machine; the workflow engine invokes the steps in sequence. This is a directed acyclic graph, or DAG: dependencies have a direction, and the workflow has no cycles. section 4.3.3
A working development pipeline is not yet production infrastructure. Production data flows must support models in the data environment for which they were trained and supply the application with the data it needs to function. Plan that transition alongside model serving rather than treating a trained model as a deployable system on its own. section 9.0 section 9.2.3
