A production data pipeline moves application data through the transformations needed to make an inference. It gathers data from operational activity and stores, creates the features and formats a model expects, and passes the result to a running model. Model serving then executes the model and produces an output. section 9.4.1
A data pipeline is a sequence of transforms managed by a workflow engine and scheduler. The workflow engine chains tasks into a directed acyclic graph (DAG): data moves forward through the specified steps without a loop. This gives the team a way to manage transformations that would otherwise be scattered across scripts and systems. section 4.3.3
Build the flow as a delivery task, not a one-time data extraction:
- Aggregate and fuse relevant sources, then transform them into records or features that models can consume. Add data tests so the results are reliable and reproducible. section 5.4
- Carry the data-delivery process from development into testing, then build its production counterpart. Reproduce the training and testing feature transformations while accounting for differences in production hardware and systems. section 9.4.1
- Document and version pipelines as code. Wrap necessary specialist tools in the standard pipeline infrastructure rather than leaving exceptional steps as undiscoverable one-off processes. section 5.4.2
- Instrument and test each step and pipeline. Report when it runs and whether it succeeds or fails; use timers or timeouts to expose operations that take too long. section 5.4.2
From an application event to an inference
A shopping-basket click can draw on the click itself, previous basket selections, user data, and commercial context. The production flow combines and processes those inputs into the model’s expected features before submitting them for inference. section 9.4.1
Figure 9.4 An abstract view of the production ML implementation. Data is pulled into and collected in a feature creation process, and then it’s submitted to a running instance of the model(s) that are used to generate an inference and an output.
Trace the boundary between collecting and preparing data, submitting features to a running model, and producing an output. section 9.4.1
