7 Decision Trees

 

He that climbs the tree has won the right to the fruit.

- Walter Scott, an 18th centenary Scottish poet.

In the last part, we explored similarity based method which was founded on the general assumption of target function’s uniform continuity; this assumption didn’t constraint our model to a specific functional form. However, in this part we start exploring a specific form of modeling assumptions, which is the form of a tree. In this chapter, we solve a problem of used cars price prediction (which is our first regression problem to encounter so far) by assuming that our target function takes the form of a binary decision tree. Moreover, we’re going to learn:

  • How to fill-in missing entries in a dataset using the rest of available entries.
  • How to train and evaluate regression models using mean squared errors.
  • How decision trees and random forests are built.
  • How to use scikit-learn to train and use decision trees and random forests.
  • What makes learning functions from data possible and what constraints it.
  • What Bias-variance Trade-off is.

7.1       Predicting the Price of a Used Car

7.1.1   Modeling the Problem with Decision Tress

7.1.2   How to Build a Decision Tree

7.1.3   Coding a Primitive Decision Tree

7.2       Training a Decision Tree with scikit-learn

7.2.1   Preparing the Data

7.2.2   Training and Evaluating the Decision Tree

7.3       Trim the Tree or Grow Yourself Forest

7.3.1   Pruning the Tree

7.3.2   Random Forests

7.4       What Controls Generalization?

7.4.1   Why do Machines Learn from Data?

7.4.2   Generalization Bounds

7.4.3   The Bias-Variance Trade-off

7.4.4   Why do Random Forests Work so Well?