Overfitting - When Fraud Detection Blocks All Transactions

Every ML model you build has one job: make good predictions on data it has never seen before. It sounds simple, but the single biggest reason models fail in production isn't bad algorithms or messy data — it's getting the balance of learning wrong. A model that learns too much from its training data becomes obsessed with noise and quirks that don't generalise. A model that learns too little never captures the real signal in the first place. Both failures have names, both are measurable, and both are fixable once you understand what's actually happening inside the model.

Overfitting and underfitting sit at opposite ends of a spectrum called the bias-variance tradeoff. Understanding this tradeoff is what separates engineers who tune models by intuition from those who tune them systematically. When you know WHY a model overfits, you stop throwing random regularisation at it and start making deliberate, principled decisions about complexity, data size, and training strategy.

By the end of this article you'll be able to plot a learning curve and diagnose whether your model is overfitting or underfitting just by looking at it. You'll have working Python code that deliberately creates both problems and then fixes them — so the concepts stick in your hands, not just your head. And you'll walk away knowing exactly which levers to pull in each scenario.

Overfitting vs. Underfitting — The Two Failure Modes of Learning

Overfitting is when a model learns the training data too precisely, including its noise and outliers, resulting in poor generalization to unseen data. Underfitting is the opposite: the model fails to capture the underlying patterns, performing poorly even on the training set. The core mechanic is a mismatch between model capacity and data complexity — too much capacity relative to signal leads to memorization; too little leads to oversimplification.

In practice, overfitting manifests as high variance: small changes in input cause large swings in predictions. Underfitting shows high bias: the model consistently misses the true relationship. A classic symptom of overfitting is training accuracy near 100% while validation accuracy plateaus or drops. Underfitting shows both training and validation accuracy stuck below acceptable thresholds. The key property is that both degrade real-world performance, but they require opposite remedies — more regularization or more data for overfitting, more capacity or better features for underfitting.

You encounter these when deploying any learned model to production. For fraud detection, an overfit model might block 99% of legitimate transactions because it memorized rare fraud patterns from training, while an underfit model might let through obvious fraud because it never learned the signal. The practical goal is to find the sweet spot where validation error is minimized — typically by monitoring the gap between training and validation metrics and using techniques like cross-validation, early stopping, or pruning.

The Technical Root: Bias vs. Variance

To fix a failing model, you must diagnose its soul. Underfitting is caused by High Bias—the model makes simplistic assumptions about the data. Overfitting is caused by High Variance—the model is overly sensitive to small fluctuations in the training set. In a production environment, we use Learning Curves (plotting Error vs. Training Set Size) to visualize this struggle.

The bias-variance tradeoff is fundamental. A model with high bias pays little attention to data and consistently underfits. A model with high variance pays too much attention to data, including noise. Finding the sweet spot is what model tuning is all about.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import learning_curve

# io.thecodeforge approach: Systematic Diagnostic
def generate_learning_curve(model, X, y):
    train_sizes, train_scores, test_scores = learning_curve(
        model, X, y, cv=5, scoring='neg_mean_squared_error'
    )
    
    # Calculate mean and standard deviation
    train_mean = -np.mean(train_scores, axis=1)
    test_mean = -np.mean(test_scores, axis=1)
    
    return train_sizes, train_mean, test_mean

# Underfitting: Linear model on non-linear data
underfit_model = LinearRegression()

# Overfitting: High-degree polynomial
overfit_model = Pipeline([
    ("poly_features", PolynomialFeatures(degree=15, include_bias=False)),
    ("std_scaler", StandardScaler()),
    ("lin_reg", LinearRegression()),
])

Detecting Overfitting Early with Validation Curves

Validation curves show how model performance changes with a hyperparameter (e.g., polynomial degree, tree depth). They help you find the sweet spot before overfitting takes hold. Plotting validation curves during development is far cheaper than discovering the problem in production.

In production systems, we automate this with CI/CD pipelines that generate validation curves for every candidate model. The pipeline rejects any model where the validation curve shows a gap larger than a configurable threshold (typically 10-15% for classification accuracy).

Here's how to generate a validation curve for polynomial degree using scikit-learn:

from sklearn.model_selection import validation_curve
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import numpy as np

def generate_validation_curve(X, y, param_range):
    model = Pipeline([
        ("poly", PolynomialFeatures(include_bias=False)),
        ("reg", LinearRegression()),
    ])
    train_scores, test_scores = validation_curve(
        model, X, y,
        param_name="poly__degree",
        param_range=param_range,
        cv=5,
        scoring="neg_mean_squared_error"
    )
    train_mean = -np.mean(train_scores, axis=1)
    test_mean = -np.mean(test_scores, axis=1)
    return param_range, train_mean, test_mean

Fixing Overfitting: Regularisation, Dropout, and Pruning

When you've confirmed overfitting, the toolkit is broad but principled. The most effective levers are:

• L1/L2 Regularisation: Adds a penalty to large weights. L1 drives weights to zero (feature selection), L2 shrinks them.
• Dropout: Randomly drops neurons during training — forces the network to learn redundant representations.
• Early Stopping: Monitors validation loss and stops training when it starts increasing.
• Reduce Model Complexity: Fewer layers, fewer trees, lower polynomial degree.
• Increase Training Data: More data reduces the impact of noise.
• Feature Selection: Remove irrelevant features that introduce noise.

Here's a production-ready Python example that applies all three regularisation techniques:

import numpy as np
from sklearn.neural_network import MLPRegressor
from tensorflow import keras
from keras import layers, regularizers, callbacks

def build_regularised_model(input_dim):
    model = keras.Sequential([
        layers.Dense(64, activation='relu', 
                     kernel_regularizer=regularizers.l2(0.01)),
        layers.Dropout(0.5),
        layers.Dense(32, activation='relu',
                     kernel_regularizer=regularizers.l2(0.01)),
        layers.Dropout(0.3),
        layers.Dense(1)
    ])
    model.compile(optimizer='adam', loss='mse')
    return model

# Early stopping callback
early_stop = callbacks.EarlyStopping(
    monitor='val_loss', patience=10, restore_best_weights=True
)

model = build_regularised_model(X_train.shape[1])
history = model.fit(X_train, y_train,
                    validation_data=(X_val, y_val),
                    epochs=200, callbacks=[early_stop], verbose=0)

Fixing Underfitting: Complexity, Features, and Training Time

Underfitting means your model is too simple. The solution is to give it more capacity. But adding complexity blindly can tip into overfitting — you need a controlled approach.

Here's a Python workflow that detects underfitting and applies fixes systematically:

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

def diagnose_and_fix_underfitting(X, y):
    # Baseline linear model
    linear_model = LinearRegression()
    linear_score = -cross_val_score(linear_model, X, y, cv=5, 
                                    scoring='neg_mean_squared_error').mean()
    
    # Try polynomial features (degree 3)
    poly_model = Pipeline([
        ('poly', PolynomialFeatures(degree=3, include_bias=False)),
        ('scaler', StandardScaler()),
        ('reg', LinearRegression())
    ])
    poly_score = -cross_val_score(poly_model, X, y, cv=5, 
                                  scoring='neg_mean_squared_error').mean()
    
    if poly_score < linear_score * 0.8:
        print("Underfitting fixed with polynomial features. Using poly model.")
        return poly_model
    else:
        print("Still underfitting. Try feature engineering or more complex algorithm.")
        return None

Production-Grade Monitoring and Alerting for Model Drift

Once your model is in production, it can still drift into overfitting or underfitting as data distributions change. This is called concept drift or covariate shift. You need automated monitoring.

Here's a Java implementation that logs warnings when drift indicators trigger:

package io.thecodeforge.ml;

import java.util.logging.Logger;
import java.time.Instant;

public class ModelHealthMonitor {
    private static final Logger logger = Logger.getLogger(ModelHealthMonitor.class.getName());
    private double baselineErrorRate;
    private double driftThreshold = 0.2; // 20% increase
    
    public ModelHealthMonitor(double baselineErrorRate) {
        this.baselineErrorRate = baselineErrorRate;
    }
    
    public void checkCurrentErrorRate(double currentErrorRate) {
        double relativeChange = (currentErrorRate - baselineErrorRate) / baselineErrorRate;
        if (relativeChange > driftThreshold) {
            logger.warning("CRITICAL: Model error rate increased by " + 
                           Math.round(relativeChange * 100) + "%. Overfitting? Underfitting? Check learning curves.");
        } else {
            logger.info("Model error rate within bounds.");
        }
    }
    
    public static void main(String[] args) {
        ModelHealthMonitor monitor = new ModelHealthMonitor(0.10);
        monitor.checkCurrentErrorRate(0.18);
    }
}

What is Underfitting? — The Model That Won't Learn

Underfitting is what happens when your model is too stupid to see the patterns right in front of it. It performs like garbage on both training and test data. This isn't a generalization problem — it's a failure to learn in the first place.

The root cause is always the same: the model's capacity is too low for the complexity of the data. You're trying to fit a straight line through a sine wave. Common culprits include: a model that's too simple (think linear regression on a non-linear problem), brutal regularization that kills signal, weak or irrelevant features, or you just didn't train long enough for the optimizer to converge.

From the bias-variance lens, underfitting is pure high bias territory. The model makes strong, dumb assumptions before seeing any data — "all relationships are linear" — and it ignores real patterns to stick with those assumptions. Variance is low because the model gives the same crappy output no matter what data you throw at it. Underfitting = High Bias + Low Variance. Memorize that.

// io.thecodeforge — ml-ai tutorial

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Generate synthetic non-linear data
np.random.seed(42)
X = np.linspace(-3, 3, 100).reshape(-1, 1)
y = np.sin(X).ravel() + 0.1 * np.random.normal(size=100)

# Deliberately underfit with a linear model
model = LinearRegression()
model.fit(X, y)

y_train_pred = model.predict(X)
print(f"Train MSE: {mean_squared_error(y, y_train_pred):.4f}")

# Both training and test errors are high
X_test = np.linspace(-3, 3, 50).reshape(-1, 1)
y_test = np.sin(X_test).ravel() + 0.1 * np.random.normal(size=50)
y_test_pred = model.predict(X_test)
print(f"Test MSE:  {mean_squared_error(y_test, y_test_pred):.4f}")

What is Overfitting? — The Model That Memorized Your Trash

Overfitting is the opposite failure mode: your model became a savant at memorizing the training data, including every outlier, every random noise spike, and every accidental duplicate. It nails training accuracy — 99% baby! — then falls apart on production data like a house of cards in a hurricane.

This happens when the model has too much capacity relative to the data. Too many parameters, too many features, too many layers, not enough regularization. The model finds patterns that don't exist. It learns the noise.

Bias-Variance breakdown: overfitting is high variance, low bias. The model makes almost no assumptions (low bias), so it contorts itself to fit every single training point. Variance is through the roof because if you train on a different sample, the model learns completely different noise patterns. Your predictions are all over the place. Overfitting = Low Bias + High Variance. If your validation loss starts going UP while training loss keeps dropping, you're here. Pull the cord.

// io.thecodeforge — ml-ai tutorial

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error

# Generate data with intentional noise features
np.random.seed(42)
n = 50
X_signal = np.random.uniform(-1, 1, (n, 2))
y = 0.8 * X_signal[:, 0] + 0.2 * X_signal[:, 1] + 0.1 * np.random.normal(size=n)

# Add 50 random noise features — classic overfitting trap
X_noise = np.random.normal(size=(n, 50))
X_full = np.hstack([X_signal, X_noise])

# Deep forest with no regularization — pure overfit
model = RandomForestRegressor(n_estimators=500, max_depth=None, min_samples_leaf=1)
model.fit(X_full, y)

# Train performance deceptively good
train_pred = model.predict(X_full)
print(f"Train MSE: {mean_squared_error(y, train_pred):.4f}")

# Test on fresh data — watch the train wreck
X_test_signal = np.random.uniform(-1, 1, (n, 2))
y_test = 0.8 * X_test_signal[:, 0] + 0.2 * X_test_signal[:, 1] + 0.1 * np.random.normal(size=n)
X_test_noise = np.random.normal(size=(n, 50))
X_test_full = np.hstack([X_test_signal, X_test_noise])
test_pred = model.predict(X_test_full)
print(f"Test MSE:  {mean_squared_error(y_test, test_pred):.4f}")

Why Every Model Needs a Holdout Set From Day One

Overfitting starts the second you evaluate on training data. You don't need complex math—you need a wall between your model and its score. A holdout set is that wall. Split before you train, not after. The test set defines reality; everything else is a lie.

In production, you can't trust a model that's never seen unseen data. Validation curves hide behind averages. Holdout exposes per-sample failure. It's the cheapest insurance you'll ever buy. If your pipeline doesn't enforce a strict 70/15/15 split at the dataset constructor, you're shipping technical debt.

Senior teams force holdout sets into CI/CD. Data leakage kills models silently. Start with a held-back slice, never touch it during tuning, and only use it for final kill/no-kill decisions. That one slice prevents months of wasted retraining.

// io.thecodeforge — ml-ai tutorial

import numpy as np
from sklearn.model_selection import train_test_split

# Simulate real-world dataset with temporal ordering
np.random.seed(42)
X = np.random.randn(1000, 10)
y = (X[:, 0] + 0.5 * np.random.randn(1000) > 0).astype(int)

# Enforce split BEFORE any preprocessing
X_train, X_holdout, y_train, y_holdout = train_test_split(
    X, y, test_size=0.15, shuffle=False
)

# Only train/val splits see tuning
X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, test_size=0.176, random_state=42
)

print(f"Train: {X_train.shape[0]} | Val: {X_val.shape[0]} | Holdout: {X_holdout.shape[0]}")

The Learning Curve Lie: Why Accuracy Metrics Hide Overfitting

Accuracy on a training set is a vanity metric. It reliably hides the silent failure of overfitting: memorization of noise rather than learning signal. You deploy a model that scores 98% accuracy on validation, and within weeks, production predictions drift, user engagement tanks, and your feature flags get blamed. The root cause is almost always a model that learned the quirks of your static holdout set instead of the true underlying patterns.

Stop trusting accuracy alone. Instead, monitor the generalization gap—the divergence between training and validation loss. A widening gap is your earliest warning that your model is memorizing, not generalizing. Apply regularization, cross-validation, or simpler architectures early. In production, track rolling accuracy windows on live data; if it dips while your eval curve looks fine, you’re seeing the lie in real time. The fix is not better data—it’s better vigilance.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

X, y = np.random.rand(1000, 20), np.random.randint(0, 2, 1000)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2)

model = RandomForestClassifier(n_estimators=100, max_depth=None)
model.fit(X_train, y_train)

train_acc = accuracy_score(y_train, model.predict(X_train))
val_acc = accuracy_score(y_val, model.predict(X_val))
gap = train_acc - val_acc

print(f"Train acc: {train_acc:.3f}")
print(f"Val acc:   {val_acc:.3f}")
print(f"Gap:       {gap:.3f}")
if gap > 0.1:
    print("WARNING: Overfitting detected — reduce model complexity.")