This chapter covers
- Why data science projects tend to fail
- What you can do when your project fails
- How to handle the negative emotions from failure
Most data science projects are high-risk ventures. You’re trying to predict something no one has predicted before, optimize something no one has optimized before, or understand data that no one has looked at before. No matter what you’re doing, you’re the first person doing it; the work is almost always exploratory. Because data scientists are continuously doing new things, you will inevitably hit a point where you find out that what you hoped for just isn’t possible. We all must grapple with our ideas not succeeding. Failure is heartbreaking and gut-wrenching; you want to stop thinking about data science and daydream about leaving the field altogether.
As an example, consider a company building a machine learning model to recommend products on the website. The likely course of events starts with some set of meetings in which the data science team convinces executives that the project is a good idea. The team believes that by using information about customers and their transactions, they can predict what customers want to buy next. The executives buy into the idea and green-light the project. Many other companies have these models, which seem straightforward, so the project should work.