DataOps keeps data-intensive systems dependable as data moves through storage, workflows, and production services. It sits between DevOps foundations and MLOps applications, managing data in motion and at rest rather than focusing only on models. section 3.4 section 3.4.1
A practical DataOps stack combines:
- Data stores, including databases, data warehouses, data lakes, and data meshes, provide governed and secured storage. section 3.4.1
- Orchestrators automate multistep workflows across resources and services; commonly discussed options include Airflow, Prefect, and Mage. section 3.4.1
- Pipelines move, clean, and define data, then run scheduled training, batch inference, and other operational jobs. section 3.4.1
- Container registries support consistent pipeline and service execution, while streaming integrations handle continuously arriving data when real-time processing is needed. section 3.4 section 3.4.1
Scheduled training workflow
An orchestrator can coordinate a pipeline that reads governed data from a store, cleans and defines it, runs a training job in a container, and supports scheduled batch inference. Streaming may be unnecessary when pseudo-real-time processing meets the user experience requirement. section 3.4.1
Figure 3.7 A high level view of an MLOps infrastructure with LLMs in mind. This diagram attempts to cover the full picture and the complexity of the many tools involved to make ML models work in production.
Use the infrastructure view to see DataOps tools around data stores, orchestrators, pipelines, streaming integrations, and container registries, alongside DevOps and MLOps systems. section 3.4
