Train Test Split & CV — The $50k Leakage Mistake

Every machine learning model you build is ultimately a gamble. You're betting that the patterns your model learned from historical data will hold up on data it's never seen — whether that's tomorrow's customer transactions, next month's medical scans, or a stock price six hours from now. If you measure your model's performance on the same data you trained it on, you're not measuring anything real. You're measuring how well it memorized the past, not how well it predicts the future.

The problem this solves has a name: data leakage and overfitting. A model that scores 99% on training data but 61% on new data hasn't learned — it's cheated. Train/test split and cross validation are the two foundational tools that force honest evaluation. They create a clear wall between what the model learns from and what it gets graded on. Without them, every accuracy score you report is fiction.

By the end of this article you'll understand exactly why naive evaluation is dangerous, how to implement a proper train/test split in scikit-learn, when to reach for K-Fold cross validation instead, and how to combine both for a production-grade evaluation pipeline. You'll also know the three mistakes that silently corrupt results for even experienced practitioners.

Why Evaluating on Training Data Is a Silent Killer

When you train a model, it adjusts its internal parameters to minimize error on the data you gave it. The more complex the model, the more it can contort itself to fit every quirk, outlier, and noise spike in that training data. A decision tree with unlimited depth will reach 100% training accuracy on almost any dataset — it just memorizes every row. That's overfitting, and it's invisible unless you test the model on data it's never touched.

Here's the subtle danger: even experienced developers fall into this trap when they tune hyperparameters. Every time you check your model's score and adjust something, you're indirectly leaking information about the test set into your decisions. This is why a held-out test set must be locked away and touched exactly once — at the very end.

Train/test split is the minimal viable defense. You take your full dataset, randomly shuffle it, and cut it into two non-overlapping pieces. The training set is the classroom. The test set is the final exam. The model never sees the exam questions until grading day. This one habit alone separates a trustworthy ML workflow from a broken one.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# io.thecodeforge: Standardizing Evaluation splits
cancer_data = load_breast_cancer()
features = cancer_data.data
targets   = cancer_data.target

# Split: 80% train, 20% test
# stratify=targets ensures class balance is maintained
X_train, X_test, y_train, y_test = train_test_split(
    features,
    targets,
    test_size=0.20,
    random_state=42,
    stratify=targets
)

# Train the model
forest_model = RandomForestClassifier(n_estimators=100, random_state=42)
forest_model.fit(X_train, y_train)

# Evaluate
train_accuracy = accuracy_score(y_train, forest_model.predict(X_train))
test_accuracy  = accuracy_score(y_test,  forest_model.predict(X_test))

print(f"Train accuracy : {train_accuracy:.4f}")
print(f"Test accuracy  : {test_accuracy:.4f}")

K-Fold Cross Validation — When One Test Split Isn't Enough

Here's the honest problem with a single train/test split: your result depends on luck. If the random split happens to put all the 'easy' examples in the test set, your model looks brilliant. If it puts all the hard ones there, your model looks terrible. With a small dataset (say, 500 rows), one unlucky split can swing your accuracy by 5–10 percentage points.

K-Fold cross validation fixes this by running the experiment K times, each time with a different fold acting as the test set. With K=5, your dataset is cut into 5 equal chunks. Round 1: train on folds 2–5, test on fold 1. Round 2: train on folds 1, 3–5, test on fold 2. And so on.

The cost is compute time — you're training K models instead of one. But the payoff is enormous: you use 100% of your data for evaluation (across all folds), and your performance estimate has far lower variance. For anything going into production, or any paper you're publishing, K-Fold is the standard.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# io.thecodeforge: Implementing robust Cross-Validation
cancer_data = load_breast_cancer()
features = cancer_data.data
targets   = cancer_data.target

# Pipeline prevents data leakage from scaler to validation fold
evaluation_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

stratified_kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

fold_scores = cross_val_score(
    evaluation_pipeline,
    features,
    targets,
    cv=stratified_kfold,
    scoring='accuracy'
)

print(f"Mean accuracy : {fold_scores.mean():.4f}")
print(f"Std deviation : {fold_scores.std():.4f}")

The Gold Standard: Train / Validation / Test and Nested CV

Once you add hyperparameter tuning to the picture, even K-Fold cross validation can leak. Here's why: if you run 50 hyperparameter combinations through the same CV folds and pick the best one, you've effectively optimised for those specific folds. The CV score of your winning model is now optimistically biased.

The production-grade solution is a three-way split: training set (model learns), validation set or inner CV (hyperparameters are tuned), and a completely held-out test set (touched exactly once for final reporting). This is often called nested cross validation — an inner loop for hyperparameter search and an outer loop for unbiased performance estimation.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

# io.thecodeforge: Production Evaluation Pipeline
cancer_data = load_breast_cancer()
features = cancer_data.data
targets   = cancer_data.target

# STEP 1: Sealed-envelope test set
X_develop, X_final_test, y_develop, y_final_test = train_test_split(
    features, targets,
    test_size=0.20,
    random_state=42,
    stratify=targets
)

# STEP 2: Pipeline construction
model_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier(random_state=42))
])

# STEP 3: GridSearch (Inner CV)
hyperparam_grid = {
    'classifier__n_estimators': [50, 100, 200],
    'classifier__max_depth':    [None, 10, 20]
}

grid_search = GridSearchCV(
    model_pipeline,
    hyperparam_grid,
    cv=StratifiedKFold(n_splits=5),
    scoring='accuracy'
)
grid_search.fit(X_develop, y_develop)

# STEP 4: Final Evaluation
best_model = grid_search.best_estimator_
print(classification_report(y_final_test, best_model.predict(X_final_test)))

Stratified Splits for Imbalanced Data — Why Random Isn't Fair

When your target classes are imbalanced — say, 95% 'no churn' and 5% 'churn' — a random split can easily create a test set with zero churn examples. Your model would appear to have 95% accuracy by simply predicting 'no churn' every time. You'd ship a completely useless model.

Stratified splitting forces each fold and each split to mirror the original class proportions. In scikit-learn, train_test_split(..., stratify=targets) and StratifiedKFold(n_splits=5) handle this for you. For regression tasks, consider StratifiedKFold by binning the target into quantiles.

For extreme imbalance (e.g., <1% minority), even stratification can be fragile. Reduce K so each fold has at least a few minority samples, or use repeated stratified splits.

from sklearn.model_selection import StratifiedKFold, train_test_split
import numpy as np

# io.thecodeforge: Stratified splits for imbalanced data
# Simulate highly imbalanced dataset (5% positive)
y = np.array([0]*950 + [1]*50)
X = np.random.randn(1000, 10)

# Without stratification – risk of test set with no minority
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(f"Unstratified test set class counts: {np.bincount(y_test)}")  # Might be [157, 3] or worse

# With stratification
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
print(f"Stratified test set class counts: {np.bincount(y_test)}")  # Should be [152, 8] (mirrors original)

# For K-Fold
skf = StratifiedKFold(n_splits=5)
for i, (train_idx, test_idx) in enumerate(skf.split(X, y)):
    print(f"Fold {i}: test class counts {np.bincount(y[test_idx])}")

Time Series Cross Validation — You Can't Use Future Data to Predict the Past

Standard K-Fold cross validation assumes data points are independent and identically distributed. For time series, that assumption is false. Observations are temporally dependent — using tomorrow's data to predict yesterday creates data leakage from the future.

Scikit-learn provides TimeSeriesSplit for exactly this situation. It uses an expanding window: training sets always precede test sets in time. Fold 1 trains on days 1-30, tests on day 31. Fold 2 trains on days 1-60, tests on day 61, and so on. This mimics how a model would be used in production — trained on past data to predict the next point.

Never use KFold or ShuffleSplit on temporal data. The random shuffle destroys the time ordering and gives you an unrealistically optimistic estimate.

import numpy as np
from sklearn.model_selection import TimeSeriesSplit
from sklearn.linear_model import LinearRegression

# io.thecodeforge: Time-series cross validation
# Simulate daily sales data (100 days)
dates = np.arange(100)
X = dates.reshape(-1, 1)  # Feature: day number
y = 2 * dates + np.random.randn(100) * 5  # Sales with trend

tscv = TimeSeriesSplit(n_splits=5)

for i, (train_idx, test_idx) in enumerate(tscv.split(X)):
    print(f"Fold {i}: train {train_idx[0]}..{train_idx[-1]}, test {test_idx[0]}..{test_idx[-1]}")
    # Train and evaluate
    model = LinearRegression().fit(X[train_idx], y[train_idx])
    score = model.score(X[test_idx], y[test_idx])
    print(f"   R^2: {score:.4f}")