Train Test Split & Cross Validation Explained — With Real Code

Every machine learning model you build is ultimately a gamble. You're betting that the patterns your model learned from historical data will hold up on data it's never seen — whether that's tomorrow's customer transactions, next month's medical scans, or a stock price six hours from now. If you measure your model's performance on the same data you trained it on, you're not measuring anything real. You're measuring how well it memorized the past, not how well it predicts the future.

The problem this solves has a name: data leakage and overfitting. A model that scores 99% on training data but 61% on new data hasn't learned — it's cheated. Train/test split and cross validation are the two foundational tools that force honest evaluation. They create a clear wall between what the model learns from and what it gets graded on. Without them, every accuracy score you report is fiction.

By the end of this article you'll understand exactly why naive evaluation is dangerous, how to implement a proper train/test split in scikit-learn, when to reach for K-Fold cross validation instead, and how to combine both for a production-grade evaluation pipeline. You'll also know the three mistakes that silently corrupt results for even experienced practitioners.

Why Evaluating on Training Data Is a Silent Killer

When you train a model, it adjusts its internal parameters to minimize error on the data you gave it. The more complex the model, the more it can contort itself to fit every quirk, outlier, and noise spike in that training data. A decision tree with unlimited depth will reach 100% training accuracy on almost any dataset — it just memorizes every row. That's overfitting, and it's invisible unless you test the model on data it's never touched.

Here's the subtle danger: even experienced developers fall into this trap when they tune hyperparameters. Every time you check your model's score and adjust something, you're indirectly leaking information about the test set into your decisions. This is why a held-out test set must be locked away and touched exactly once — at the very end.

Train/test split is the minimal viable defense. You take your full dataset, randomly shuffle it, and cut it into two non-overlapping pieces. The training set is the classroom. The test set is the final exam. The model never sees the exam questions until grading day. This one habit alone separates a trustworthy ML workflow from a broken one.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# io.thecodeforge: Standardizing Evaluation splits
cancer_data = load_breast_cancer()
features = cancer_data.data
targets   = cancer_data.target

# Split: 80% train, 20% test
# stratify=targets ensures class balance is maintained
X_train, X_test, y_train, y_test = train_test_split(
    features,
    targets,
    test_size=0.20,
    random_state=42,
    stratify=targets
)

# Train the model
forest_model = RandomForestClassifier(n_estimators=100, random_state=42)
forest_model.fit(X_train, y_train)

# Evaluate
train_accuracy = accuracy_score(y_train, forest_model.predict(X_train))
test_accuracy  = accuracy_score(y_test,  forest_model.predict(X_test))

print(f"Train accuracy : {train_accuracy:.4f}")
print(f"Test accuracy  : {test_accuracy:.4f}")

K-Fold Cross Validation — When One Test Split Isn't Enough

Here's the honest problem with a single train/test split: your result depends on luck. If the random split happens to put all the 'easy' examples in the test set, your model looks brilliant. If it puts all the hard ones there, your model looks terrible. With a small dataset (say, 500 rows), one unlucky split can swing your accuracy by 5–10 percentage points.

K-Fold cross validation fixes this by running the experiment K times, each time with a different fold acting as the test set. With K=5, your dataset is cut into 5 equal chunks. Round 1: train on folds 2–5, test on fold 1. Round 2: train on folds 1, 3–5, test on fold 2. And so on.

The cost is compute time — you're training K models instead of one. But the payoff is enormous: you use 100% of your data for evaluation (across all folds), and your performance estimate has far lower variance. For anything going into production, or any paper you're publishing, K-Fold is the standard.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# io.thecodeforge: Implementing robust Cross-Validation
cancer_data = load_breast_cancer()
features = cancer_data.data
targets   = cancer_data.target

# Pipeline prevents data leakage from scaler to validation fold
evaluation_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

stratified_kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

fold_scores = cross_val_score(
    evaluation_pipeline,
    features,
    targets,
    cv=stratified_kfold,
    scoring='accuracy'
)

print(f"Mean accuracy : {fold_scores.mean():.4f}")
print(f"Std deviation : {fold_scores.std():.4f}")

The Gold Standard: Train / Validation / Test and Nested CV

Once you add hyperparameter tuning to the picture, even K-Fold cross validation can leak. Here's why: if you run 50 hyperparameter combinations through the same CV folds and pick the best one, you've effectively optimised for those specific folds. The CV score of your winning model is now optimistically biased.

The production-grade solution is a three-way split: training set (model learns), validation set or inner CV (hyperparameters are tuned), and a completely held-out test set (touched exactly once for final reporting). This is often called nested cross validation — an inner loop for hyperparameter search and an outer loop for unbiased performance estimation.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

# io.thecodeforge: Production Evaluation Pipeline
cancer_data = load_breast_cancer()
features = cancer_data.data
targets   = cancer_data.target

# STEP 1: Sealed-envelope test set
X_develop, X_final_test, y_develop, y_final_test = train_test_split(
    features, targets,
    test_size=0.20,
    random_state=42,
    stratify=targets
)

# STEP 2: Pipeline construction
model_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier(random_state=42))
])

# STEP 3: GridSearch (Inner CV)
hyperparam_grid = {
    'classifier__n_estimators': [50, 100, 200],
    'classifier__max_depth':    [None, 10, 20]
}

grid_search = GridSearchCV(
    model_pipeline,
    hyperparam_grid,
    cv=StratifiedKFold(n_splits=5),
    scoring='accuracy'
)
grid_search.fit(X_develop, y_develop)

# STEP 4: Final Evaluation
best_model = grid_search.best_estimator_
print(classification_report(y_final_test, best_model.predict(X_final_test)))

Frequently Asked Questions