Lecture 9: Decision Trees and Ensemble Methods

What this lecture covers

This lecture introduces decision trees as one of the first non-linear model classes in the course, after weeks spent on linear models like logistic regression and SVMs. It works through how a tree recursively partitions the input space, what loss functions make good splits, and how trees extend to regression and categorical features. The second half explains why trees alone tend to overfit, then shows how ensembling, bagging, random forests and boosting turn that weakness into one of the most competitive model families in practice.

By the end, you should be able to explain how a decision tree greedily picks splits, why cross-entropy loss is preferred over misclassification loss for choosing splits, and how bagging and boosting attack the bias-variance tradeoff from opposite directions. This sets up the ensemble methods (random forests, AdaBoost, gradient boosting) that show up frequently in applied machine learning.

Key ideas

• Recursive partitioning: a decision tree builds a partition of the input space top-down and greedily, asking one yes/no question about a feature and threshold at each step.
• Split loss: a split is chosen to minimize the combined loss of the two resulting child regions relative to the parent region.
• Misclassification loss is too insensitive: it can fail to distinguish a clearly better split from a clearly worse one, because it only looks at the majority class in each region.
• Cross-entropy and Gini loss: both are strictly concave functions of the class proportions, so any split (other than a useless one) provably reduces loss, which is why they are preferred over misclassification loss for choosing splits.
• Regression trees: leaves predict the mean of the target values in that region instead of a majority class, using squared error as the loss.
• Regularizing trees: heuristics such as minimum leaf size, maximum depth, and pruning after growing a full tree control the high variance of unrestricted trees; stopping early on "minimum decrease in loss" is discouraged because useful splits can look weak until combined with a later split.
• Bagging (bootstrap aggregation): training many models on bootstrap-resampled versions of the training set and averaging their predictions reduces variance, at a small cost in bias, by decorrelating the individual models.
• Random forests and boosting: random forests add extra decorrelation by restricting each split to a random subset of features; boosting instead reduces bias by additively combining weak, high-bias models (such as depth-1 decision stumps) trained on reweighted versions of the data.

Walkthrough

Decision trees as recursive partitioning (0:42)

Using a running example of predicting whether you can ski given a month and a latitude, the lecture shows why a linear classifier struggles with this data (the positive region is split into disconnected patches) while a decision tree handles it naturally. A tree partitions the space top-down and greedily: at each region, it picks the feature and threshold that best separates the data, then recursively repeats the process on each resulting sub-region, formalized as choosing a split (j, t) that divides a parent region R_p into two children based on whether feature j is below or above threshold t.

Choosing splits: misclassification vs. cross-entropy loss (15:24)

The lecture defines the misclassification loss of a region as one minus the proportion of the majority class, then shows a worked example where two different splits produce the same total misclassification loss even though one split is intuitively much better. This motivates cross-entropy loss, borrowed from information theory, which sums class proportions times their logarithm. A geometric argument shows that because cross-entropy (and the related Gini loss) is a strictly concave function of class proportion, any real split reduces the loss, whereas the misclassification loss's piecewise-linear shape can fail to register a gain even when one exists.

Regression trees, categorical variables and regularization (31:05)

Trees extend naturally to regression by predicting the mean target value at each leaf and using squared error as the loss. Categorical features are handled by splitting on subsets of categories rather than thresholds, though the number of possible subsets grows exponentially with the number of categories (efficient exact solutions exist for binary classification). Because an unconstrained tree can grow until every leaf holds a single example, trees are naturally high variance and prone to overfitting; the lecture covers regularization heuristics such as minimum leaf size, maximum depth or node count, and warns against stopping early based on minimum loss decrease, since a weak first split can still be part of a strong combination. Growing a full tree and then pruning back using a validation set is presented as the more reliable approach.

Runtime and the limits of a single tree (43:28)

With n examples, f features, and tree depth d, test-time prediction costs O(d), and training costs O(nfd), which is fast in practice because d is typically bounded by log(n). The lecture then names the tree's core weaknesses: it lacks additive structure, so a simple diagonal decision boundary (easy for logistic regression) requires many splits to approximate; and because of this plus its high variance, a single tree usually has lower predictive accuracy than other model classes despite being fast, interpretable, and able to handle categorical variables directly.

Bagging: bootstrap aggregation (1:00:11)

Ensembling is motivated by a basic variance identity: averaging independent, identically distributed variables divides variance by the number of variables, while for correlated variables the variance of the mean depends on both the correlation rho and the count. Since collecting many new training sets or many different algorithms is impractical, bagging instead draws bootstrap samples (samples of size n drawn with replacement from the training set, treating the training set itself as a stand-in for the population), trains one model per bootstrap sample, and averages their predictions. Adding more bootstrap models drives down variance without materially increasing overfitting, though bootstrap samples remain correlated with each other because they are all drawn from the same underlying training set, which places a lower bound on how much variance bagging can remove. Because decision trees are already low-bias, high-variance models, they are an especially good fit for bagging.

Random forests and boosting (1:12:46)

Random forests extend bagged decision trees by restricting each split to consider only a random subset of features, further decorrelating the trees beyond what bootstrapping alone achieves. This matters when one feature is a very strong predictor: without the restriction, every bagged tree would tend to split on it first, keeping the trees highly correlated. Boosting works in the opposite direction, reducing bias instead of variance by additively combining a sequence of weak, high-bias models such as depth-one decision stumps. Each stump is trained on a reweighted version of the data that up-weights previously misclassified examples, and each stump's vote is weighted (roughly by its accuracy) in the final combined classifier G(x) = sum of alpha_m * G_m(x). This is the mechanism behind AdaBoost and, with variations, gradient boosting methods such as XGBoost.

Before you watch

• Be comfortable with logistic regression and SVMs, since the lecture contrasts trees against these linear models throughout.
• Review basic variance and covariance identities for sums of random variables, which underpin the ensembling arguments.
• Familiarity with bias-variance tradeoff from earlier in the course will make the bagging-versus-boosting comparison easier to follow.

Check your understanding

1. Why can misclassification loss fail to distinguish between a better split and a worse one, and how does cross-entropy loss fix this?
2. How does a decision tree's prediction rule differ between classification and regression at a leaf node?
3. Why is bootstrapping able to reduce variance without a large increase in overfitting as more models are added?
4. What is the purpose of restricting each split in a random forest to a random subset of features?
5. How does boosting's approach to combining weak models differ from bagging's approach, in terms of which part of the bias-variance tradeoff each one targets?