Chapter 2: What is MLOps?

Introduction

The chapter introduction frames MLOps around the gap between a useful model and a reliable ML system, then previews the iterative life cycle and the chapter’s scope.

• Business value and infrastructure gap: Explain why building a model is often not the hardest part of delivering ML value: organizations must connect business requirements to the data, training, deployment, service integration, and ongoing accuracy needed in production.
  Why it matters: This gap explains why MLOps is needed even when an existing model architecture can be reused.
• Continuous ML improvement loop: Describe the introduction’s view that ML often creates value as part of a closed loop that improves continuously, rather than as a one-time model-development effort.
  Why it matters: The loop is the framework for the life-cycle stages that follow.
• Chapter focus and comparisons: Describe the chapter’s stated focus on input requirements, problem definition, and design philosophies that influence model design, followed by comparisons with DevOps and discussion of organizational MLOps maturity.
  Why it matters: This roadmap helps place the chapter’s later comparisons and maturity levels in context.

Key terms: business value, requirements, infrastructure, MLOps, closed loop, continuously improves, ML life cycle, mental map, input requirements, problem definition

2.1 The iterative MLOps life cycle

This section presents ML engineering as an evolving loop and explains why repeatable workflows and stakeholder agreement are needed before and throughout a project.

• Evolving ML loop: Explain why a real-world ML system is better understood as a closed loop than as a finished model: data, models, hyperparameters, and surrounding conditions change over time, so changes in one part affect the others.
  Why it matters: The loop helps you anticipate how changes will affect system design and maintenance.
• Repeatable model change workflow: Justify a robust workflow for building, testing, and deploying frequent model changes: it preserves developer velocity and standardizes work across teams. Explain why separately tracking model artifacts and code, including their lineage, matters as versions evolve.
  Why it matters: A repeatable, traceable workflow lets teams improve models without making each rollout a new deployment problem.
• Collaborative problem definition: Describe how the ML loop begins with a well-defined problem aligned across business and product, technical, and legal and compliance stakeholders before data collection and modeling. Give examples of the questions each group raises and explain how collaboration clarifies scope, success criteria, and potential challenges.
  Why it matters: A precise, jointly agreed problem makes technical decisions and evaluation serve the intended business goal.

Key terms: closed loop, iterative MLOps life cycle, ephemeral, robust workflow, developer velocity, standardization, lineage, model (weights + networks), code (model architecture and supporting code), problem statement

2.1.1 Data collection

This section explains how careful, traceable data collection supports the iterative ML life cycle and introduces the datasets used in the book’s projects.

• Data collection quality criteria: Explain how to collect data that is relevant, sufficiently large and high quality, diverse, and representative of the deployment environment while avoiding harmful bias and sample leakage. Apply these criteria to the car price estimator by selecting second-hand passenger cars and sampling the target market appropriately.
  Why it matters: Problems in collected data carry forward into every later modeling step.
• Research and enterprise data differences: Distinguish researchers’ frequent use of reviewed open-source datasets from enterprise practitioners’ need to collect domain-specific data or fine-tune on deployment-environment data, while noting that both face the same underlying data-quality requirements.
  Why it matters: An available benchmark dataset may not reflect the conditions a production model will face.
• Data lineage and versioning: Explain that data lineage tracks the origin and versions of raw, intermediate, augmented, and annotated data, including when, where, how, and why it was collected. Describe how this makes dataset revisions and error correction easier, such as recreating a dataset with a versioned ETL pipeline rather than rerunning complex extraction scripts.
  Why it matters: Tracking data from the start makes inevitable dataset revisions more manageable.
• Project datasets and labeling: Describe the section’s contrasting project data: MIDV-500 video clips for ID document detection, optional human image annotation with CVAT versus procedurally labeled synthetic data, and MovieLens 1M user, movie, and rating files that will be split to simulate incoming production data and versioned.
  Why it matters: These examples show how collection and labeling challenges vary by data modality and project.

Key terms: data relevance, class bias, unintentional leakage, representative of the deployment environment, diversity, open source datasets, fine-tuned, transfer learning, deployment environment, data lineage

2.1.2 Exploratory Data Analysis

This section explains how exploratory data analysis reveals data problems, tests assumptions, and guides repeated decisions throughout the MLOps life cycle.

• EDA diagnostic questions: Describe how EDA examines the available data to uncover problems and estimate the effort needed to develop and maintain a model. Give examples of questions about schema and invalid values, cleaning and outliers, distributions and class balance, or feature quality and cost.
  Why it matters: These questions can expose problems before they become expensive modeling or production failures.
• Multivariate pattern discovery: Explain how examining multiple variables together can reveal distributions and patterns useful for robust features, and when dimensionality reduction methods may help.
  Why it matters: Relationships among variables can be more informative than inspecting each one separately.
• Data assumptions as risks: Explain why assumptions about data and its distribution must be validated, and describe putting checks early in the data pipeline to detect violations. Give an example such as incoming schema, sampling bias, or the meaning of a field.
  Why it matters: Early checks can narrow down failures quickly and make it possible to change approach before investing too much work.
• Iterative EDA and production context: Describe why EDA is revisited as new insights emerge or experiments fail, rather than treated as a one-time step. Explain how its findings can help ML engineers investigate root causes and deployment problems such as drift, or prompt a change in modeling approach.
  Why it matters: Revisiting the data helps teams respond when experiments or production behavior challenge their original view of the problem.

Key terms: exploratory data analysis (EDA), schema, class balancing, features, outliers, multivariate analysis, dimensionality reduction, Principal Component Analysis (PCE), t-distributed Stochastic Neighbor Embedding (t-SNE), assumptions

2.1.3 Modeling and training

This section explains how model choice, traceable training practices, and modular code support reproducible experimentation, then illustrates those ideas across the book’s ML projects.

• Model choice and objectives: Explain that modeling starts by choosing an algorithm suited to the original prediction objective, considering the input data, problem type, expected performance, and required metrics.
  Why it matters: A model is useful only if it fits the problem and can meet the intended objectives.
• Training lineage and traceability: Explain how model and data versioning, experiment tracking, and training pipelines reduce manual mistakes and make training results traceable and reproducible. Distinguish tying a model to a dataset version from logging an experiment’s results, configuration, data, and checkpoints.
  Why it matters: Traceability lets a team reproduce, debug, compare, and report on models rather than lose track of how results were produced.
• Modular configuration and search: Describe a modular codebase in which a trained model is an artifact produced from code, data, and configurations, with configurations tracked in Git. Explain how configurable architectures and training hyperparameters enable faster experiments and automated hyperparameter search, while starting simple avoids overparametrization.
  Why it matters: Separating and tracking these inputs makes experiments easier to change, automate, and debug.
• Project training examples: Describe how this modeling step appears in the section’s projects: fine-tuning YOLOv8 on labeled identity-card images, starting the recommender with a baseline before retraining and tracking, and introducing embeddings, vector databases, and chat interfaces for RAG while reusing core MLOps ideas.
  Why it matters: The examples show that training choices differ by application even when traceability and iteration remain useful.

Key terms: modeling phase, problem statement, classification, detection, regression, model and data versioning, experiment tracking, model training pipelines, lineage, traceability

2.1.4 Model evaluation

This section explains how to evaluate a trained model for its target domain using appropriate metrics, a controlled holdout dataset, error analysis, and repeatable evaluation workflows.

• Domain aligned evaluation metrics: Explain how evaluation metrics should follow the problem definition and target domain rather than training-test performance alone. Describe how task type, class imbalance, drift, and the costs of different errors affect metric choice.
  Why it matters: A model can score well on a metric yet fail to meet the needs of its intended use.
• Production representative holdout evaluation: Describe automated evaluation on a separate holdout dataset curated to resemble production data, with safeguards against leakage. Explain why the holdout and data splits require lineage, versioning, and tracking as they evolve.
  Why it matters: A controlled, representative holdout makes evaluation more trustworthy and repeatable.
• Error analysis and feedback: Explain how examining misclassified samples for patterns, systemic problems, and bias feeds findings back into data collection. Describe testing sensitivity and robustness with adversarial tests or edge-case simulations.
  Why it matters: Evaluation should reveal failures to address, not merely produce a score.
• Repeatable evaluation workflow: Explain why multistage evaluation benefits from automated pipelines, centralized result tracking, documentation, and visibility as its code, data, and components evolve.
  Why it matters: A visible evaluation history avoids repeated work and supports comparisons as models change.
• Optional model interpretability: Describe interpretability as an evaluation practice some organizations require, involving inspection or visualization of model behavior or techniques such as LIME and SHAP. Explain why its benefits should be weighed against implementation complexity before adopting it.
  Why it matters: Interpretability can expose model limitations, but it is not presented as a mandatory starting step.

Key terms: target domain, precision, recall, F1 score, mean absolute error (MAE), mean squared error (MSE), holdout dataset, data leakage, lineage, versioning

2.1.5 Deployment

Deployment turns a trained model into a production service or device-based application, with testing, staged release, version control, and monitoring to keep it useful.

• Model deployment destinations: Describe the two deployment approaches in this section: serving a model through an API endpoint in a microservices architecture, or running it on specific hardware or an edge device.
  Why it matters: Where a model runs shapes how it is packaged and evaluated.
• Deployment optimization and testing: Explain why a trained model is often optimized for its target deployment environment and why performance must be tested on that final, optimized version rather than inferred from training results.
  Why it matters: Testing the version that actually runs helps prevent misleading performance expectations.
• Staging and production transitions: Trace the two-stage release described here: deploy a model to staging for evaluation against the production model, then move it to production after evaluation. Explain how version control and lineage record transitions and make rollback to an earlier model simpler.
  Why it matters: A staged, traceable release makes model changes easier to assess and reverse.
• Book project deployment choices: State how the section says the book will deploy its projects: BentoML serves the first two projects over HTTP, Docker packages them, Yatai deploys them, and Chainlit supplies a chat interface for the final RAG application.
  Why it matters: These choices connect the deployment ideas to the systems built later in the book.
• Monitoring after model release: Explain that deployment continues after a model goes live: production models need monitoring for business value, while BentoML services expose metrics such as requests per second and HTTP response codes because every prediction cannot be evaluated.
  Why it matters: A running endpoint alone does not show whether the deployed model remains useful.

Key terms: API endpoint, microservices architecture, edge device, optimization, latency, quantization, deployment environment, staging, production, model version control

2.1.6 Monitoring

This section explains how monitoring keeps a deployed model reliable by detecting data and performance problems, investigating their causes, and alerting people who can act.

• Purpose of production monitoring: Explain why a deployed model needs ongoing monitoring: its performance can change as inputs or concepts change or as hardware or software fails, and wrong predictions can harm the business.
  Why it matters: Deployment creates value only if the model continues to behave reliably in production.
• Three monitoring variants: Distinguish data monitoring of input anomalies and drift, performance monitoring of production metrics over time, and error monitoring of wrong predictions and edge cases.
  Why it matters: Different monitoring signals reveal different ways a production model can go wrong.
• Diagnostic records and model versions: Describe how model version control supplies context about a model and its training or dataset, while logging helps reproduce edge cases and investigate systemic issues or code changes; together they provide documentation in regulated industries.
  Why it matters: Records help turn a detected problem into one that can be traced and understood.
• Actionable alerts and retraining: Explain why reliable alerts must reach a responsible person, not merely be recorded, and describe the section's example of using detected drift to trigger retraining runs.
  Why it matters: Monitoring provides little protection if its findings never lead to corrective action.

Key terms: model monitoring, downstream metrics, concept drift, data monitoring, data drift, performance monitoring, error monitoring, model version control, logging, edge cases

2.1.7 Maintenance, updates, and review

This section explains how monitoring feeds maintenance and updates throughout the ML life cycle, then previews how the book’s projects will put that feedback into practice.

• Monitoring driven corrective actions: Explain how monitoring outputs close the ML loop by prompting bug fixes, collection of data for edge cases, and responses to drift that lead to new data collection and retraining.
  Why it matters: Monitoring is useful when its findings lead to changes that keep a production model effective.
• Maintenance throughout the loop: Describe why maintenance is not confined to a final stage: deployment replaces a production model with a newer version, while data collection changes in response to concept or data drift noticed through monitoring.
  Why it matters: Thinking of maintenance across the loop helps connect monitoring results to the components that must actually change.
• Project maintenance scenarios: Describe how the ID card detector and recommender projects will use monitoring, drift detection, and evaluation when new data or model choices arise: an unfamiliar card tests the detector as a possible replacement, while additional user data and a new architecture prompt comparison with the production recommender using defined metrics.
  Why it matters: These examples show how feedback informs a controlled choice about what model to deploy.
• Public facing RAG safeguards: State that the RAG pipeline’s maintenance focus shifts toward LLM-specific observability with Langfuse and toward safeguards and tests such as guardrails, adversarial testing, input sanitation, prompt safety, and hallucination testing for a public-facing application.
  Why it matters: A public-facing RAG system calls for visibility into its behavior as well as checks on unsafe or unreliable responses.

Key terms: maintenance, updates, and review, model monitoring, edge cases, data drift, retrain, deployment component, data collection, concept and data drift, newer version, ID card detection project

2.2 Why is robust MLOps important ?

This section explains why production ML is difficult and why investing in robust MLOps pays off despite its complexity.

• Interconnected ML system complexity: Explain how each stage of the ML loop adds complexity, and how choices about component integration, data exchange, and architecture affect model output and the whole project.
  Why it matters: A model delivers consistent business value only when the surrounding system works together.
• Data stewardship and governance: Describe why robust data pipelines must balance scale and visibility with privacy, security, fairness, and bias concerns, and why compliance also calls for explainable, interpretable ML.
  Why it matters: Data failures and governance lapses can undermine a project even when its model performs well.
• Cross-functional communication gaps: Explain why specialists and stakeholders need a shared language despite using different tools and success measures, such as model quality metrics versus deployment and serving metrics.
  Why it matters: The ML loop depends on people at different stages being able to collaborate, not just on their separate expertise.
• Production performance and scaling: Explain why a strong evaluation result does not guarantee production value: optimization can change performance, and serving heavy traffic requires characterization, testing, and architecture beyond toy workloads.
  Why it matters: A model that is too slow or fails under load cannot deliver its promised business value.
• Real-world data failure detection: Explain how messy inputs, edge cases, or improper data can cause visible failures or silently skew outputs, and how monitoring and data validation reduce—but do not eliminate—that risk.
  Why it matters: Silent bad outputs are particularly difficult to catch once a model meets real-world data.
• Tooling tradeoffs and long-term investment: Justify a robust MLOps strategy despite fragmented tooling: comprehensive tools may leave gaps, specialist tools must be combined to fit team needs, and quick single-model solutions can accumulate technical debt.
  Why it matters: Up-front investment supports long-term maintainability, delivery speed, and business value.

Key terms: ML loop, system architecture, data exchange, data pipelines, visibility, privacy, data security, fairness, diversity, precision

2.3 Role of MLOps in a mature organization

This section explains how a shared MLOps strategy helps a mature organization experiment faster, use resources efficiently, collaborate, and operate reliable models at scale.

• Faster model experimentation: Explain how MLOps infrastructure lets data scientists focus on business problems and iterate quickly by abstracting infrastructure and supporting versioning, comparison, lineage, and A/B testing.
  Why it matters: These capabilities make it easier to test ideas and respond to changing business needs.
• Shared organizational ML infrastructure: Explain why a cohesive ML strategy improves costs, collaboration, and scaling compared with teams building separate workflows and infrastructure.
  Why it matters: Shared components and coordinated resource management prevent duplicated effort and help models handle production demands.
• Automated and repeatable workflows: Describe how MLOps and CI/CD automate repetitive ML life-cycle tasks, reducing human error and making results more deterministic while freeing data scientists for higher-value work.
  Why it matters: Automation makes routine work more dependable without consuming the time needed for modeling and innovation.
• Traceable models and default practices: Explain how monitoring and lineage make changing models easier to debug, review, and roll back, and how integrating practices such as logging into everyday tooling encourages their use by default.
  Why it matters: When good practices are built into the workflow, teams can identify problems and recover without reconstructing a model's history by hand.

Key terms: infrastructure abstraction, model versioning, lineage, A/B testing, central ML workflow, shared components, resource allocation, automated scaling, collaboration, MLOps

2.4 DevOps vs. MLOps

This section explains what MLOps shares with DevOps and why data, evolving models, and experimental ML workflows require additional practices.

• Shared DevOps and MLOps practices: Describe how both DevOps and MLOps use automation, CI/CD, and collaboration across roles to make delivery faster and more reliable.
  Why it matters: These shared practices explain why MLOps builds on DevOps rather than replacing it.
• Data and evolving model lifecycle: Explain why MLOps must manage data and models beyond the software and infrastructure life cycle emphasized by DevOps. Describe how models can change without code changes, creating a need for Continuous Training and model management and versioning.
  Why it matters: Data and model changes are central reasons ML systems need practices beyond conventional software delivery.
• Model monitoring and governance: Explain why MLOps monitors model performance and addresses interpretability, bias, and compliance. Connect possible performance degradation to changes in input data.
  Why it matters: A deployed model can become less effective as its inputs change, even when the software keeps running.
• Experimental workflow tracking: Describe MLOps as experimental and iterative, and explain why it tracks experiments rather than only software builds.
  Why it matters: Tracking experiments preserves the history of an iterative modeling process that build records alone would miss.

Key terms: automation, CI/CD, multidisciplinary, data life cycle, Continuous Training, model management, versioning, model interpretability, bias, monitoring

2.5 Levels of MLOps maturity

The section introduces automation of the ML loop as a measure of MLOps maturity and a driver of faster model development and adaptation.

• Automation and MLOps maturity: Explain that the extent to which a company automates the steps of the ML loop indicates its MLOps maturity and affects how quickly it can develop new models or adapt to changes.
  Why it matters: This gives you a way to interpret the maturity levels that follow.

Key terms: level of automation, ML loop, MLOps maturity, velocity

2.5.1 Level 0: Basic

Level 0 is the starting MLOps maturity stage, marked by manual model workflows, no monitoring, and infrequent releases.

• Level 0 MLOps characteristics: Describe Level 0 as a common starting point for companies beginning MLOps, where scripts and manual work drive model building, training, and deployment, models are not monitored, and releases are infrequent.
  Why it matters: These characteristics provide a baseline for comparing later MLOps maturity levels.

Key terms: Level 0, manual script-based processes, monitoring, release iterations

2.5.2 Level 1: Intermediate

Level 1 MLOps replaces model-only deployment with an automatically triggered retraining pipeline, supported by reusable components and operational safeguards.

• Continuous retraining pipeline: Explain how Level 1 moves beyond deploying only a model: it deploys a retraining pipeline that responds to new data and continuously delivers updated models to prediction services.
  Why it matters: This shift from a deployed model to a deployed pipeline defines Level 1 maturity.
• Reusable components and symmetry: Describe how freely shared, modular pipeline components let teams use the same pipeline for deployment in development and production, an arrangement called experimental-operational symmetry.
  Why it matters: Reusing one pipeline across environments connects experimentation with production operation.
• Level 1 enabling components: State the components the section identifies as important for Level 1: robust data and model validation, a feature store if needed, pipeline monitoring with lineage and metadata, and an ML pipeline that runs on custom triggers.
  Why it matters: These components make continuous retraining manageable rather than merely automatic.

Key terms: Level 1, retraining pipeline, continuous delivery of models, prediction services, modular pipeline components, experimental-operational symmetry, data and model validation, feature store, pipeline monitoring, lineage

2.5.3 Level 2: Advanced

This section describes advanced MLOps maturity, explains how to use maturity levels to plan progress, and connects those ideas to the platform-building work ahead.

• Advanced pipeline automation: Describe how Level 2 automates the building and deployment of pipelines themselves and makes pipeline components a critical part of the ML loop, while data analysis and model analysis remain largely unautomated.
  Why it matters: This distinguishes advanced MLOps from simply automating individual ML tasks.
• Shared component ownership: Explain that pipelines and components at Level 2 belong to the whole team or organization rather than only ML engineers, and that data scientists can propose new components based on their observations.
  Why it matters: Advanced maturity changes who contributes to the ML workflow, not just how much of it is automated.
• Maturity levels as planning guide: Explain that the maturity levels are a rough guide rather than equally difficult linear steps, and that locating an organization's current level helps plan steady progress and build an MLOps culture. State that the book's tools can scale across all maturity levels.
  Why it matters: The levels are most useful for choosing next steps, not treating maturity as a rigid checklist.

Key terms: Level 2: Advanced, pipelines, pipeline components, ML loop, data analysis, model analysis, components, ML engineers, data scientists, levels of maturity

2.6 Summary

The chapter summary frames MLOps as an iterative, business-driven practice and explains why its challenges require disciplined processes and organizational learning.