Overfitting and Underfitting in ML — Causes, Detection and Fixes

Every ML model you build has one job: make good predictions on data it has never seen before. It sounds simple, but the single biggest reason models fail in production isn't bad algorithms or messy data — it's getting the balance of learning wrong. A model that learns too much from its training data becomes obsessed with noise and quirks that don't generalise. A model that learns too little never captures the real signal in the first place. Both failures have names, both are measurable, and both are fixable once you understand what's actually happening inside the model.

Overfitting and underfitting sit at opposite ends of a spectrum called the bias-variance tradeoff. Understanding this tradeoff is what separates engineers who tune models by intuition from those who tune them systematically. When you know WHY a model overfits, you stop throwing random regularisation at it and start making deliberate, principled decisions about complexity, data size, and training strategy.

By the end of this article you'll be able to plot a learning curve and diagnose whether your model is overfitting or underfitting just by looking at it. You'll have working Python code that deliberately creates both problems and then fixes them — so the concepts stick in your hands, not just your head. And you'll walk away knowing exactly which levers to pull in each scenario.

The Technical Root: Bias vs. Variance

To fix a failing model, you must diagnose its soul. Underfitting is caused by High Bias—the model makes simplistic assumptions about the data. Overfitting is caused by High Variance—the model is overly sensitive to small fluctuations in the training set. In a production environment, we use Learning Curves (plotting Error vs. Training Set Size) to visualize this struggle.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import learning_curve

# io.thecodeforge approach: Systematic Diagnostic
def generate_learning_curve(model, X, y):
    train_sizes, train_scores, test_scores = learning_curve(
        model, X, y, cv=5, scoring='neg_mean_squared_error'
    )
    
    # Calculate mean and standard deviation
    train_mean = -np.mean(train_scores, axis=1)
    test_mean = -np.mean(test_scores, axis=1)
    
    return train_sizes, train_mean, test_mean

# Underfitting: Linear model on non-linear data
underfit_model = LinearRegression()

# Overfitting: High-degree polynomial
overfit_model = Pipeline([
    ("poly_features", PolynomialFeatures(degree=15, include_bias=False)),
    ("std_scaler", StandardScaler()),
    ("lin_reg", LinearRegression()),
])

Production-Grade Implementation: Handling Model Persistence

When deploying models at TheCodeForge, we ensure that the model complexity is handled during the build phase. Below is a Java representation of a model metadata service that flags potential overfitting based on the gap between training and validation accuracy.

package io.thecodeforge.ml;

import java.util.logging.Logger;

public class ModelGuard {
    private static final Logger logger = Logger.getLogger(ModelGuard.class.getName());
    private static final double OVERFIT_THRESHOLD = 0.15; // 15% gap

    public static void validateModelPerformance(double trainAcc, double valAcc) {
        double gap = Math.abs(trainAcc - valAcc);
        
        if (trainAcc < 0.60 && valAcc < 0.60) {
            logger.warning("CRITICAL: Model is UNDERFITTING. High Bias detected.");
        } else if (gap > OVERFIT_THRESHOLD) {
            logger.warning("CRITICAL: Model is OVERFITTING. High Variance detected. Gap: " + gap);
        } else {
            logger.info("Model state is HEALTHY. Generalization optimal.");
        }
    }

    public static void main(String[] args) {
        // Example: High Training Accuracy, Low Validation
        validateModelPerformance(0.98, 0.72);
    }
}

Frequently Asked Questions