Chapter 4: Designing reliable ML systems

Introduction

This introduction maps the tools in a small ML platform, focusing the chapter on MLflow experiment tracking and Feast feature storage while previewing deployment and monitoring.

• MLflow experiment and model management: Explain how exploratory analysis, train/test splitting, and trials with different features, hyperparameters, and models lead to selecting a model; distinguish MLflow experiment tracking from storing the selected model and its dependencies in the MLflow model registry.
  Why it matters: Tracking trials and preserving the chosen model make experiments easier to compare, reproduce, and hand off.
• Feast feature store role: Explain why engineered features must be registered and how Feast provides an interface to store and access them for training and predictions.
  Why it matters: Keeping features accessible supports both collaboration and use of the model after experimentation.
• Deployment and monitoring roadmap: Describe the proposed path after models and features are stored: Kubeflow Pipelines for batch predictions, BentoML for a real-time API, and Evidently for monitoring shifts in feature-data distribution. State that this chapter focuses on experiment tracking and the feature store, using an income-band binary classifier as its example.
  Why it matters: This roadmap shows how the chapter’s tools fit into a broader production ML workflow.

Key terms: exploratory data analysis (EDA), MLflow experiment tracker, MLflow model registry, hyperparameters, feature engineering, feature store, Feast, registering features, Kubeflow Pipelines, batch

4.1 MLflow for experiment tracking

This section explains why ML experiments need tracking, why a shared MLflow tracking server is preferable to ad hoc records, and how the chapter will introduce it through an income-classification example.

• Reasons for experiment tracking: Explain how recording data, model settings, and results supports reproducibility, choosing among models, and comparing a retrained model with earlier versions.
  Why it matters: Tracking makes it possible to explain which model performed best and reproduce or compare results later.
• Shared MLflow tracking server: Describe the drawbacks of using tagged version-control commits or a separate parameter file, and explain how MLflow's centralized tracking server records experiment parameters, data, metrics, and artifacts for teams to compare and replicate runs.
  Why it matters: A shared server gives collaborators a structured place to find the information behind an experiment.
• Income classification walkthrough: Describe the planned move from exploratory work in a Jupyter Notebook to income-data EDA, several classification models, and MLflow tracking logic for those experiments.
  Why it matters: The income example connects the need for tracking to the hands-on work that follows.

Key terms: reproducibility, model selection, performance tracking, MLflow, tracking server, experiment parameters, hyperparameters, metrics, artifacts, Jupyter Notebook

4.1.1 Data exploration

The section explores an income dataset with Pandas summaries and plots showing how categorical variables relate to a binary income target.

• Income data descriptive summaries: Explain how Pandas info and describe are used to inspect the income data: info reports about 30,000 rows, while describe reports each categorical variable’s unique values, most frequent value, and its frequency.
  Why it matters: These summaries establish what the dataset contains before comparing variables with income.
• Categorical income distribution plots: Describe plotting categorical-variable distributions against the binary income target (<=50K or >50K), saving the plots in categorical_variable_plots, and reading the workclass example as showing that self-employed people are more likely to earn over $50K.
  Why it matters: The plots make relationships between categories and the income target visible.

Key terms: Pandas info, Pandas describe, categorical variables, top, frequency, target variable, <=50K, >50K, categorical_variable_plots, Workclass

4.1.2 MLflow tracking

This section shows how to track exploratory plots, datasets, models, parameters, and metrics in MLflow, using both manual logging and autologging.

• Experiments runs and artifacts: Explain how to start a local MLflow tracking server, set an experiment in a notebook, and record plots in a run. Describe how experiments contain runs and how the UI displays each run’s artifacts alongside its other logged information.
  Why it matters: This structure lets you find the outputs and results of a particular experiment execution.
• Prepared datasets and external storage: Describe how the example encodes categorical features and the income target before splitting the data into training and test sets. Explain why it saves datasets in MinIO and records their external locations for reproducibility or debugging.
  Why it matters: A tracked result is more useful when you can recover the data used to produce it.
• Manual model and dataset logging: Trace how the decision-tree example logs dataset references, evaluation metrics, model parameters, and the trained model in an MLflow run. Distinguish the datasets saved in MinIO from the dataset metadata and external paths logged in MLflow.
  Why it matters: Logging the data reference and model results together makes a training run easier to inspect and reproduce.
• Autologging benefits and limits: Explain how enabling XGBoost autologging records the model, parameters, and plots without separate logging calls. State why the example still uses manual logging for custom metrics or a preferred dataset format.
  Why it matters: Autologging saves tracking code, but its defaults do not replace every explicit log entry.

Key terms: tracking server, experiment, run, run ID, artifacts, mlflow.log_artifacts, pd.get_dummies, one-hot encoding, train_test_split, MinIO

4.1.3 MLflow model registry

This section uses MLflow metrics to select and register the best model, then explains how the registry supports reuse and what must be addressed before production.

• Metric based model selection: Explain how logged test AUC metrics are used to compare the decision tree, random forest, and XGBoost runs, either by filtering and viewing runs in the MLflow UI or by querying them with the MLflow client.
  Why it matters: A recorded metric gives the team a clear basis for choosing which model to register.
• Client registration workflow: Trace how the MLflow client finds the experiment and its runs, obtains the selected run's model URI, and registers that model under a name.
  Why it matters: This turns a selected experiment result into a model that can be found and reused.
• Registry lifecycle and team access: Describe what the MLflow model registry is for, including version tracking, governance and reproducibility, retrieval for inference, and possible promotion through staging and production; explain why team collaboration needs an accessible MLflow installation rather than the local instance in the example.
  Why it matters: Registering a model is useful only if its versions can be managed and the people who need it can access it.
• Feature consistency handoff: Explain why registering the model is not the final step before production: its features must also be registered so training, retraining, and inference use appropriate, consistently engineered features.
  Why it matters: A reusable model still needs reliable access to the features it expects.
• Prompt management tool distinction: State that MLflow has a Prompts tab but the section describes it as limited, and that later chapters use Langfuse for more robust prompt tracking, evaluation, and versioning.
  Why it matters: Prompt management is related to model management but may call for a different tool.

Key terms: MLflow UI, MLflow client, metrics.roc_auc_score_test, chart view, random forest model, get_experiment_by_name, search_runs, run ID, model URI, register_model

4.2 Feast as a feature store

The section uses an income-classifier example to show how Feast organizes reusable features, retrieves historically correct values, and serves features for training and real-time inference.

• Shared income classifier features: Describe how the income classifier’s single feature file is split into demographic, relationship, and occupation files, each with a timestamp and user ID. Explain why putting features in a common location separates feature generation from modeling and makes features reusable.
  Why it matters: Shared features reduce duplicated preparation work across models and teams.
• Point-in-time feature retrieval: Explain that retrieving features for a user at a given time means choosing the most recent recorded values no later than that time. Describe how Feast supplies this point-in-time-correct retrieval instead of requiring custom joining logic.
  Why it matters: A historically correct feature set prevents a model from using values recorded after the time of the prediction or training example.
• Offline and online feature stores: Trace how a feature pipeline populates the offline store, Feast materializes features into a low-latency online store, and the model uses offline features for training and online features for real-time inference. Describe the registry’s feature definitions and the Python SDK’s access to both stores.
  Why it matters: This design makes the same defined features available for training while supporting fast retrieval during live predictions.

Key terms: demographic features, relationship features, occupation features, MinIO, feature generation, point-in-time correct feature sets, feature retrieval, Feast feature store, offline store, online store

4.2.1 Registering features

This section shows how to define user features and their storage in Feast, then register those definitions with Feast apply.

• User entity definition: Explain that a Feast entity identifies what the features describe: here, a user identified by a unique user ID. Describe the entity definition’s name, description, and string data type.
  Why it matters: The entity connects feature values to the user they describe.
• MinIO feature source locations: Describe how each feature category is linked to files in MinIO using an S3 path and an S3 endpoint override pointing to the locally port-forwarded MinIO service.
  Why it matters: Feast needs a reachable location for the files that supply the features.
• Feature views and TTL: Explain that a FeatureView logically groups features and specifies its name, entity, feature list, file source, and TTL, with optional tags. Describe how TTL limits the historical lookback and why this example uses 365 days.
  Why it matters: The view defines both which features belong together and how old a feature Feast may use in a historical dataset.
• Feature store configuration: Describe what feature_store.yaml configures: the project and registry location, a Filestore offline store for files in MinIO, and a Redis online store with its connection string. Explain that the registry is the shared location for feature and entity definitions.
  Why it matters: These settings tell Feast where definitions live and which stores serve offline and online features.
• Feast registration sequence: Trace the final registration steps: port-forward MinIO and Redis to the local environment, then run Feast apply to update the registry with entity and FeatureView definitions. State that configuring the repository alone has not yet registered those definitions.
  Why it matters: The definitions become available in the feature registry only after the required connections are available and apply runs.

Key terms: entity, user_id, entity.py, MinIO, s3 path, s3 endpoint override, port-forward, FeatureView, ttl, timedelta

4.2.2 Retrieving features

This section shows how to retrieve historical features for training or batch inference and the latest online features for real-time inference using Feast.

• Historical feature retrieval: Explain how to retrieve historical features with Feast: initialize a feature store using the path to feature_store.yaml, then call get_historical_features with an entity DataFrame and a list of requested features. Describe the required user_id and event_timestamp columns and the demographic:Sex feature example.
  Why it matters: Historical retrieval supplies features associated with an entity and timestamp for training or batch inference.
• Online feature materialization and retrieval: Describe how feast materialize copies data within a specified START_TS–END_TS interval from the offline store to the online store before get_online_features retrieves the latest feature for a user_id. Distinguish this real-time inference use from historical retrieval for training or batch inference.
  Why it matters: Real-time inference needs current features available in the online store, not a historical training lookup.

Key terms: get_historical_features, feature_store.yaml, user_id, event_timestamp, demographic:Sex, training_df, feast materialize, START_TS, END_TS, offline store

4.2.3 Feature server

This section shows how Feast’s feature server exposes API endpoints for retrieving online features without using a Feast SDK.

• Feature server access: Explain that Feast’s feature server provides API endpoints for interacting with the feature store, especially for applications written in languages without a Feast SDK.
  Why it matters: The API lets applications access features even when they cannot use a Feast SDK.
• Online feature retrieval: Describe starting a local feature server with `feast serve`, then using a `curl` request that specifies desired features and a `user_id` to retrieve values from the online feature store. State that the response includes feature and entity names with their values.
  Why it matters: This is the section’s concrete way to retrieve online features through the server.

Key terms: Feast, feature server, API endpoints, Feast SDK, feast serve, curl, user_id, online feature store

4.2.4 Using the Feast UI

The section shows how to inspect registered features in the Feast UI and explains why MLflow and Feast still need pipeline automation for a reliable production system.

• Feast UI registry inspection: Describe how to access the Feast UI and what it shows about registered features, entities, and feature data sources, including the ability to filter feature views by tags.
  Why it matters: The UI provides a quick way to inspect what the feature registry contains and where features come from.
• Tracking features and automation gap: Explain the distinct roles of MLflow and Feast in a reliable ML system and why they do not eliminate manual work. Describe how the next chapter proposes to automate retraining and feature updates with Kubeflow pipelines.
  Why it matters: Tracking models and managing features are important foundations, but production reliability also requires repeatable automation.

Key terms: Feast UI, feature registry, feature views, entities, data sources, tags, MLflow experiment tracker, model registry, Feast, training and inference

4.3 Summary

This section summarizes how MLflow tracks and manages models and how Feast manages features for training and inference.