11 Features and feature engineering
This chapter covers
- The iterative process of feature engineering
- Analyzing feature importance
- Selecting appropriate features for your model
- Pros and cons of feature stores
It is often said that a mediocre model with great features will outperform a great model with poor features. From our experience, this statement couldn’t be more true. Features are the critical inputs for your system; they drive your algorithms, provide essential patterns for the model, and feed the data it needs to learn and make predictions. Without good features, the model is blind, deaf, and dumb.
While the role of feature engineering is not crucial for a system designed with a deep learning core in mind, no machine learning (ML) practitioner can ignore their role. In a sense, framing some fancy multimodal data into a deep learning model or even making a prompt for a large language model is a specific way of feature engineering, and that’s why classic feature-related techniques like feature importance analysis are still very relevant.
This chapter explores the art and science of creating effective features. We will cover tools that help determine the most valuable features for the system, the engineering challenges we can face, the factors and tradeoffs we should consider while selecting the right subset of features, and how we can ensure that the selected features are reliable and robust.