Decision Trees Explained: How They Split, Score and Overfit

Every time a bank decides whether to approve your loan, or a doctor's diagnostic tool flags a high-risk patient, or a streaming service labels content as inappropriate — there's a good chance a decision tree is somewhere in that pipeline. They're not flashy, but they're the backbone of some of the most reliable ML systems in production, and they're the building block of powerhouses like Random Forest and XGBoost.

The problem decision trees solve is deceptively simple: given a pile of labelled examples, figure out a set of rules that correctly categorises new, unseen examples. The magic is in HOW those rules are chosen. A bad algorithm might split data arbitrarily. A decision tree uses mathematical criteria — Gini impurity or information gain — to always pick the split that creates the purest, most separable groups. That's what gives it predictive power.

By the end of this article you'll understand exactly how a tree chooses where to split (and why that maths matters), how to train and visualise one in Python with real data, how to diagnose and fix overfitting with pruning and depth control, and what to say when an interviewer asks you to compare Gini impurity to entropy on the spot.

What is Decision Trees?

Decision Trees is a core concept in ML / AI. Rather than starting with a dry definition, let's see it in action and understand why it exists.

// TheCodeForge — Decision Trees example
// Always use meaningful names, not x or n
public class ForgeExample {
    public static void main(String[] args) {
        String topic = "Decision Trees";
        System.out.println("Learning: " + topic + " 🔥");
    }
}

Frequently Asked Questions