11 Features and feature engineering

 

This chapter covers

  • The iterative process of feature engineering
  • Analyzing feature importance
  • Selecting appropriate features for your model
  • Pros and cons of feature stores

It is often said that a mediocre model with great features will outperform a great model with poor features. From our experience, this statement couldn’t be more true. Features are the critical inputs for your system; they drive your algorithms, provide essential patterns for the model, and feed the data it needs to learn and make predictions. Without good features, the model is blind, deaf, and dumb.

While the role of feature engineering is not crucial for a system designed with a deep learning core in mind, no machine learning (ML) practitioner can ignore their role. In a sense, framing some fancy multimodal data into a deep learning model or even making a prompt for a large language model is a specific way of feature engineering, and that’s why classic feature-related techniques like feature importance analysis are still very relevant.

This chapter explores the art and science of creating effective features. We will cover tools that help determine the most valuable features for the system, the engineering challenges we can face, the factors and tradeoffs we should consider while selecting the right subset of features, and how we can ensure that the selected features are reliable and robust.

11.1 Feature engineering: What are you?

11.1.1 Criteria of good and bad features

11.1.2 Feature generation 101

11.1.3 Model predictions as a feature

11.2 Feature importance analysis

11.2.1 Classification of methods

11.2.2 Accuracy–interpretability tradeoff

11.2.3 Feature importance in deep learning

11.3 Feature selection

11.3.1 Feature generation vs. feature selection

11.3.2 Goals and possible drawbacks

11.3.3 Feature selection method overview

11.4 Feature store

11.4.1 Feature store: Pros and cons

11.4.2 Desired properties of a feature store

11.4.3 Feature catalog

11.5 Design document: Feature engineering

11.5.1 Features for Supermegaretail

11.5.2 Features for PhotoStock Inc.

Summary