Logistic Regression — The Threshold That Missed 3% Cancers

Every day, your email provider quietly decides whether to drop a message into your inbox or your spam folder. Your bank flags a transaction as fraud or lets it through. A hospital algorithm predicts whether a tumour is malignant or benign. All of these are binary decisions — yes or no, 0 or 1 — and Logistic Regression is one of the most reliable, interpretable, and battle-tested tools for making them. It's been doing this job since the 1950s and it's still the first model data scientists reach for when the stakes are high and the explanation matters.

The core problem Logistic Regression solves is one that Linear Regression cannot: predicting a bounded probability. If you used ordinary linear regression to classify emails, nothing stops it from predicting a 'spam probability' of 2.7 or -0.4 — which is meaningless. Logistic Regression wraps its output in a sigmoid function that mathematically constrains every prediction to live between 0 and 1, giving you an actual probability you can act on.

By the end of this article you'll understand not just how to call LogisticRegression().fit() in scikit-learn, but why the sigmoid function exists, what the coefficients are actually telling you about the real world, how to tune the decision threshold for different business goals, and exactly what questions an interviewer will ask you to separate the practitioners from the people who just skimmed a tutorial.

Why Logistic Regression Is a Linear Classifier, Not a Regression

Logistic regression predicts the probability that an input belongs to a binary class by passing a linear combination of features through the logistic (sigmoid) function. The core mechanic: compute z = w·x + b, then output σ(z) = 1 / (1 + e⁻ᶻ), which squashes any real number into a (0,1) probability. Despite the name, it's a classification algorithm — the 'regression' refers to fitting a linear decision boundary, not predicting a continuous value.

Training maximizes log-likelihood via gradient descent, not least squares. The loss function is cross-entropy: -[y log(ŷ) + (1-y) log(1-ŷ)]. This penalizes confident wrong predictions heavily — a 0.97 probability on a false positive costs far more than 0.51. The decision threshold is a separate hyperparameter; default 0.5 is rarely optimal. In production, you tune this threshold using precision-recall or ROC curves, not accuracy alone.

Use logistic regression when you need interpretable probabilities, fast inference, or a strong baseline. It's the go-to for binary classification on linearly separable or near-separable data — spam detection, churn prediction, medical diagnosis. It scales to millions of features with L1/L2 regularization and trains in minutes on a single machine. For non-linear boundaries, add feature crosses or kernel tricks, but know that deep nets will outperform once data exceeds ~100k examples with complex interactions.

The Sigmoid Function — Why Logistic Regression Uses This Specific Curve

Linear Regression gives you a straight line. That's great for predicting house prices, but terrible for predicting probabilities — because a straight line extends to infinity in both directions and probability must stay between 0 and 1.

The sigmoid function (also called the logistic function, which is where the algorithm gets its name) is the mathematical fix. Its formula is σ(z) = 1 / (1 + e^(-z)). Feed it any real number — whether it's -1000 or +1000 — and it maps the output to the range (0, 1). Large positive inputs push the output close to 1. Large negative inputs push it close to 0. Right at zero, you get exactly 0.5.

The input z is itself a linear combination of your features: z = w₀ + w₁x₁ + w₂x₂ + ... — exactly like Linear Regression. So Logistic Regression is really just Linear Regression with its output passed through the sigmoid. That single design decision makes the output interpretable as a probability, which is the foundation everything else builds on.

The model learns the weights (w values) by maximising the likelihood that the predicted probabilities match the actual labels in your training data — a process called Maximum Likelihood Estimation, optimised via gradient descent.

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    """The core of logistic regression — maps any real number to (0, 1)."""
    return 1 / (1 + np.exp(-z))

# Create a range of z values to visualise the S-curve
z_values = np.linspace(-10, 10, 300)
probabilities = sigmoid(z_values)

# Annotate key points so the behaviour is obvious
key_points = {
    -5: sigmoid(-5),   # Very likely class 0
     0: sigmoid(0),    # Exactly on the decision boundary
     5: sigmoid(5),    # Very likely class 1
}

print("=== Sigmoid Output at Key Z-Values ===")
for z, prob in key_points.items():
    label = "→ class 1" if prob >= 0.5 else "→ class 0"
    print(f"  z = {z:+d}  |  P(y=1) = {prob:.4f}  {label}")

# Plot the S-curve
plt.figure(figsize=(8, 4))
plt.plot(z_values, probabilities, color='steelblue', linewidth=2.5, label='σ(z)')
plt.axhline(y=0.5, color='tomato', linestyle='--', linewidth=1.5, label='Decision boundary (0.5)')
plt.axvline(x=0,   color='gray',   linestyle=':',  linewidth=1.2)
plt.fill_between(z_values, probabilities, 0.5,
                 where=(probabilities >= 0.5), alpha=0.12, color='steelblue', label='Predict class 1')
plt.fill_between(z_values, probabilities, 0.5,
                 where=(probabilities < 0.5),  alpha=0.12, color='tomato',    label='Predict class 0')
plt.xlabel('z  (linear combination of features)')
plt.ylabel('Predicted Probability')
plt.title('The Sigmoid Function — How Logistic Regression Converts Scores to Probabilities')
plt.legend()
plt.tight_layout()
plt.savefig('sigmoid_curve.png', dpi=150)
print("\nPlot saved to sigmoid_curve.png")

Training on Real Data — Breast Cancer Classification End-to-End

Theory only sticks when you see it on real data. We'll use scikit-learn's built-in Breast Cancer dataset — 569 tumour samples, each described by 30 numeric features (mean radius, texture, smoothness, etc.), labelled as malignant (0) or benign (1). The goal is to predict the label from the measurements.

There are a few things to get right here that tutorials often skip. First, feature scaling matters enormously for Logistic Regression because gradient descent converges far faster when all features live on a similar scale. If 'mean area' is in the thousands and 'mean fractal dimension' is near 0.05, the loss surface is elongated and training is sluggish. StandardScaler fixes this.

Second, you should always look at your model's coefficients after training. Each coefficient tells you how much the log-odds of the positive class change for a one-unit increase in that feature. A large positive coefficient means that feature is a strong predictor of benign; a large negative one means it predicts malignant. That interpretability is exactly why doctors, banks and regulators often prefer Logistic Regression over a black-box neural network — you can explain every decision.

Third, accuracy alone is a dangerous metric for medical data. A model that predicts 'benign' for every sample gets ~63% accuracy on this dataset without learning anything. Always check precision, recall and the confusion matrix.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score
)

# ── 1. Load Data ──────────────────────────────────────────────────────────────
cancer_data = load_breast_cancer()
feature_matrix = cancer_data.data        # Shape: (569, 30)
target_labels  = cancer_data.target      # 0 = malignant, 1 = benign
feature_names  = cancer_data.feature_names

print(f"Dataset shape : {feature_matrix.shape}")
print(f"Class balance : {np.bincount(target_labels)} (malignant, benign)")

# ── 2. Train / Test Split ─────────────────────────────────────────────────────
# stratify= ensures both splits keep the same class ratio
(X_train, X_test,
 y_train, y_test) = train_test_split(
    feature_matrix, target_labels,
    test_size=0.20,
    random_state=42,
    stratify=target_labels
)

# ── 3. Feature Scaling — critical for gradient-descent-based models ───────────
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # fit only on training data!
X_test_scaled  = scaler.transform(X_test)         # apply same scale to test

# ── 4. Train the Model ───────────────────────────────────────────────────────
# max_iter=1000 because the default 100 often hits a ConvergenceWarning
logistic_model = LogisticRegression(max_iter=1000, random_state=42)
logistic_model.fit(X_train_scaled, y_train)

# ── 5. Predict & Evaluate ────────────────────────────────────────────────────
y_pred_labels = logistic_model.predict(X_test_scaled)
y_pred_proba  = logistic_model.predict_proba(X_test_scaled)[:, 1]  # P(benign)

print("\n=== Confusion Matrix ===")
cm = confusion_matrix(y_test, y_pred_labels)
print(f"  True Negatives  (Malignant correctly caught) : {cm[0,0]}")
print(f"  False Positives (Malignant missed as Benign) : {cm[0,1]}")
print(f"  False Negatives (Benign wrongly flagged)     : {cm[1,0]}")
print(f"  True Positives  (Benign correctly caught)    : {cm[1,1]}")

print("\n=== Classification Report ===")
print(classification_report(y_test, y_pred_labels,
                            target_names=['Malignant', 'Benign']))

roc_auc = roc_auc_score(y_test, y_pred_proba)
print(f"ROC-AUC Score : {roc_auc:.4f}")

# ── 6. Inspect Coefficients — this is where Logistic Regression shines ───────
print("\n=== Top 5 Features Pushing Towards Malignant (negative coefficients) ===")
coef_pairs = sorted(
    zip(feature_names, logistic_model.coef_[0]),
    key=lambda pair: pair[1]
)
for feature_name, coefficient in coef_pairs[:5]:
    print(f"  {feature_name:<35}  coef = {coefficient:+.4f}")

print("\n=== Top 5 Features Pushing Towards Benign (positive coefficients) ===")
for feature_name, coefficient in coef_pairs[-5:][::-1]:
    print(f"  {feature_name:<35}  coef = {coefficient:+.4f}")

Tuning the Decision Threshold — When 0.5 Is the Wrong Cut-Off

Most tutorials treat the 0.5 threshold as sacred. It isn't. The threshold is a business decision, not a mathematical constant, and understanding when to move it separates good practitioners from great ones.

Consider the breast cancer case: a False Negative (predicting benign when the tumour is actually malignant) sends a patient home without treatment. A False Positive (flagging benign as malignant) means an unnecessary biopsy — uncomfortable, but survivable. These mistakes are not equal. You should tolerate more False Positives to drive False Negatives toward zero, which means lowering your threshold below 0.5 so the model cries 'malignant' sooner.

Conversely, in a spam filter, a False Positive (blocking a legitimate email) is worse than a False Negative (letting spam through). Here you'd raise the threshold.

The ROC curve plots True Positive Rate against False Positive Rate across every possible threshold. The area under it (AUC-ROC) tells you how well the model separates classes regardless of threshold — it's the metric to optimise during model selection. The Precision-Recall curve is more informative when your classes are heavily imbalanced.

The code below shows how to find the threshold that maximises recall for malignant detection — exactly the kind of analysis you'd run before deploying a medical model.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_recall_curve, roc_curve
import matplotlib.pyplot as plt

# ── Reuse the trained model setup from the previous example ──────────────────
cancer_data    = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    cancer_data.data, cancer_data.target,
    test_size=0.20, random_state=42, stratify=cancer_data.target
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled  = scaler.transform(X_test)
logistic_model = LogisticRegression(max_iter=1000, random_state=42)
logistic_model.fit(X_train_scaled, y_train)

# Predicted probabilities for the positive class (benign = 1)
y_proba_benign = logistic_model.predict_proba(X_test_scaled)[:, 1]

# ── Find threshold that maximises recall for MALIGNANT class ─────────────────
# Note: precision_recall_curve works with respect to the positive label.
# We flip the probabilities so 'malignant' becomes the positive class.
y_proba_malignant = 1 - y_proba_benign
y_test_malignant  = 1 - y_test        # 1 = malignant, 0 = benign (flipped)

precisions, recalls, thresholds = precision_recall_curve(
    y_test_malignant, y_proba_malignant
)

# We want recall >= 0.99 with the highest possible precision
high_recall_mask = recalls[:-1] >= 0.99   # exclude last point (no threshold)
candidates = list(zip(
    thresholds[high_recall_mask],
    precisions[:-1][high_recall_mask],
    recalls[:-1][high_recall_mask]
))

print("=== Threshold Candidates Achieving ≥99% Recall on Malignant Class ===")
print(f"  {'Threshold':>12}  {'Precision':>10}  {'Recall':>8}")
for thresh, prec, rec in candidates:
    print(f"  {thresh:>12.4f}  {prec:>10.4f}  {rec:>8.4f}")

# Pick the threshold with highest precision among our high-recall candidates
best_threshold, best_precision, best_recall = max(candidates, key=lambda t: t[1])
print(f"\n✔ Best threshold = {best_threshold:.4f}")
print(f"  At this threshold — Precision: {best_precision:.4f}, Recall: {best_recall:.4f}")

# ── Apply the chosen threshold and see its real-world impact ─────────────────
# We predict 'malignant' whenever P(malignant) >= best_threshold
y_pred_tuned = (y_proba_malignant >= best_threshold).astype(int)

malignant_actual   = np.sum(y_test_malignant == 1)
malignant_caught   = np.sum((y_pred_tuned == 1) & (y_test_malignant == 1))
malignant_missed   = malignant_actual - malignant_caught

print(f"\n=== Clinical Impact at Tuned Threshold ===")
print(f"  Total malignant tumours in test set : {malignant_actual}")
print(f"  Correctly flagged (True Positives)  : {malignant_caught}")
print(f"  Missed (False Negatives)            : {malignant_missed}  ← the dangerous ones")

# ── ROC Curve ─────────────────────────────────────────────────────────────────
fpr, tpr, roc_thresholds = roc_curve(y_test_malignant, y_proba_malignant)

plt.figure(figsize=(6, 5))
plt.plot(fpr, tpr, color='steelblue', lw=2, label='ROC Curve')
plt.plot([0, 1], [0, 1], color='gray', linestyle='--', label='Random classifier')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate (Recall)')
plt.title('ROC Curve — Malignant Detection')
plt.legend()
plt.tight_layout()
plt.savefig('roc_curve.png', dpi=150)
print("\nROC curve saved to roc_curve.png")

Maximum Likelihood Estimation and Log-Loss — How Logistic Regression Learns

You've seen the sigmoid and the coefficients. But how does the model actually find those coefficients? The answer is Maximum Likelihood Estimation (MLE). Logistic Regression doesn't minimise squared error (like Linear Regression does) — it maximises the probability of seeing the observed data given the parameters.

Mathematically, MLE finds the weights w that maximise the product of predicted probabilities for each training sample. For a binary classification task, this product is:

L(w) = ∏ P(y=1 | x)^y · (1 - P(y=1 | x))^{(1-y)}

Taking the logarithm turns the product into a sum, which is easier to optimise. The negative of that sum is called log-loss (binary cross-entropy). The model uses gradient descent to minimise log-loss. This is why Logistic Regression uses log-loss instead of MSE: log-loss is convex with respect to the weights, which guarantees that gradient descent will find the global optimum.

Convexity matters because it means you're never stuck in a local minimum. With MSE and sigmoid, the loss surface has hills and valleys — gradient descent can get trapped. Log-loss is a smooth bowl shape. That's the mathematical guarantee you need for a stable training process.

In scikit-learn, you don't see this — it's wrapped inside the fit() method. But understanding the loss function is crucial for debugging: if your loss is not decreasing smoothly, check the learning rate (not exposed in sklearn's default LogisticRegression, but you control it via tol and max_iter) or consider a different solver.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import log_loss

# Use the same breast cancer data
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    cancer.data, cancer.target, test_size=0.2, random_state=42, stratify=cancer.target
)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Train model with different regularisation strengths
for C in [0.01, 0.1, 1, 10, 100]:
    model = LogisticRegression(C=C, max_iter=1000, random_state=42)
    model.fit(X_train, y_train)
    y_pred_train_proba = model.predict_proba(X_train)[:, 1]
    y_pred_test_proba = model.predict_proba(X_test)[:, 1]
    train_loss = log_loss(y_train, y_pred_train_proba)
    test_loss = log_loss(y_test, y_pred_test_proba)
    acc = model.score(X_test, y_test)
    print(f"C={C:>6.2f} | Train log-loss: {train_loss:.4f} | Test log-loss: {test_loss:.4f} | Test Acc: {acc:.4f}")

# Observe: as C increases (less regularisation), train loss decreases, test loss may start increasing (overfitting).

Regularisation — L1 (Lasso) and L2 (Ridge) in Logistic Regression

Logistic Regression without regularisation can overfit, especially when you have many features or highly correlated predictors. Regularisation adds a penalty term to the loss function that discourages large coefficients. Scikit-learn's LogisticRegression uses L2 regularisation by default (controlled by the C parameter).

L2 (Ridge) adds the squared sum of coefficients to the loss. It shrinks all coefficients toward zero but rarely makes them exactly zero. Use L2 when you expect all features to contribute some signal, or when features are correlated (it handles multicollinearity gracefully).

L1 (Lasso) adds the absolute sum of coefficients. It can drive some coefficients to exactly zero, performing automatic feature selection. Use L1 when you have many irrelevant features and want a sparse model. The trade-off: L1 can be unstable with highly correlated features — it might pick one and drop the other arbitrarily.

ElasticNet combines L1 and L2 penalties. In scikit-learn, you can use LogisticRegression with penalty='elasticnet' and set the l1_ratio parameter. This gives you the best of both worlds: sparsity from L1 and stability from L2.

The C parameter controls the inverse of regularisation strength. Lower C = more regularisation (simpler model). Tune C via cross-validation — too high C leads to overfitting, too low C underfits. This is the most important hyperparameter to tune for Logistic Regression.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.preprocessing import StandardScaler

cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    cancer.data, cancer.target, test_size=0.2, random_state=42, stratify=cancer.target
)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Compare L1 and L2 with same C
models = {
    'L2 (default)': LogisticRegression(penalty='l2', C=1.0, solver='lbfgs', max_iter=1000),
    'L1 (lasso)': LogisticRegression(penalty='l1', C=1.0, solver='saga', max_iter=1000),
    'ElasticNet (l1_ratio=0.5)': LogisticRegression(penalty='elasticnet', C=1.0, solver='saga', l1_ratio=0.5, max_iter=1000),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    nonzero_coefs = np.sum(np.abs(model.coef_) > 1e-10)
    test_acc = model.score(X_test, y_test)
    print(f"{name:20} | Non-zero coefficients: {nonzero_coefs:2d} | Test accuracy: {test_acc:.4f}")

# Output shows L1 produces sparser models (fewer non-zero coefficients).

Feature Importance — Why Coefficients Tell You More Than Accuracy

You trained a logistic regression. The confusion matrix looks good. Now what? If you deploy without understanding which features actually drive the decision, you're flying blind. Logistic regression gives you something most black-box models don't: interpretable coefficients.

The sign tells you direction. Positive coefficient means higher feature values push probability toward class 1. The magnitude tells you impact — but only after scaling. If you've got age in years and income in dollars, raw coefficients are incomparable. Standardise your features first, or use odds ratios.

Odds ratio = exp(coef). An odds ratio of 2.0 means a one-unit increase in that feature doubles the odds of the positive class. This is how you explain to a product manager why "hours worked per week" matters more than "education level" for predicting income >$50K. They don't care about log-odds. They care about actionable levers.

Production teams waste weeks tuning hyperparameters when the real insight is in the coefficients. Read them. Read them before you touch the decision threshold.

// io.thecodeforge — ml-ai tutorial

import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

df = pd.read_csv('adult_income.csv')
features = ['age', 'hours_per_week', 'education_years', 'marital_status']
X = df[features]
y = df['income_above_50k']

# Scale or coefficients are meaningless
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)

model = LogisticRegression(C=1.0, penalty='l2')
model.fit(X_train, y_train)

odds_ratios = np.exp(model.coef_[0])
for name, or_val in zip(features, odds_ratios):
    print(f'{name:20s} odds_ratio={or_val:.3f}, coef={model.coef_[0][i]:.3f}')

Multicollinearity — The Silent Killer of Coefficient Stability

Logistic regression assumes your features are independent. In the real world, they're not. Hours worked per week and income? Correlated. Education years and job type? Correlated. When two features carry similar information, the model distributes coefficient weight between them unpredictably.

This isn't just a stats textbook problem. I've seen a production model flip coefficient signs across retraining runs because age and years_of_experience had a correlation of 0.89. One week age was positive, the next it was negative. The model's accuracy stayed the same, but every stakeholder lost trust.

Diagnose it with Variance Inflation Factor (VIF). VIF > 5 means that feature is heavily explained by other features. VIF > 10 means you're in trouble. Drop one of the correlated features, combine them into a ratio, or use L1 regularisation (Lasso) which can zero out one of them.

Don't trust feature importance from a logistic regression with multicollinearity. Trust the VIF scores first.

// io.thecodeforge — ml-ai tutorial

import pandas as pd
import numpy as np
from statsmodels.stats.outliers_influence import variance_inflation_factor
from sklearn.linear_model import LogisticRegression

df = pd.read_csv('adult_income.csv')
features = ['age', 'hours_per_week', 'education_years', 'marital_status', 'years_of_experience']
X = df[features].dropna()

# Add constant for intercept
X_with_const = np.column_stack([np.ones(X.shape[0]), X])

vif_data = pd.DataFrame()
vif_data['feature'] = ['const'] + features
vif_data['VIF'] = [variance_inflation_factor(X_with_const, i) for i in range(X_with_const.shape[1])]

print(vif_data)

# Remove high-VIF feature and retrain
X_reduced = df[['age', 'hours_per_week', 'education_years', 'marital_status']]
model = LogisticRegression()
model.fit(X_reduced, df['income_above_50k'])
print('Coefficients after removing years_of_experience:')
print(dict(zip(X_reduced.columns, model.coef_[0])))

Model Building in Scikit-learn — Why Defaults Won't Save You

You don't need a PhD to fit a logistic regression in scikit-learn. You need to know which knobs to turn and why the defaults will stab you in production.

First, the class_weight parameter. Default is 'None', which assumes your classes are balanced. Real-world fraud or churn datasets? You'll have 99% negative, 1% positive. Without class_weight='balanced', your model learns to predict everything negative and hits 99% accuracy while catching zero fraud. Senior engineers catch this before the pipeline breaks.

Second, solver choice. 'lbfgs' is the modern default — fast, handles L2 regularization, converges reliably on small-to-medium data. For high-dimensional sparse data (think NLP with 50k features), switch to 'saga' — it supports L1 penalty and multiclass multinomial. Never use 'liblinear' unless your data is tiny; it's a noob trap.

Third, C is the inverse of regularization strength. C=1.0 is default, but you should cross-validate between 0.01 and 100. Why? Because your feature scales matter. If one feature is purchase_amount (range $1 to $10k) and another is click_count (0 to 50), the default regularization penalizes the smaller feature unfairly. Scale your data with StandardScaler before fitting, or watch your coefficients lie to you.

// io.thecodeforge — ml-ai tutorial

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

# Real patient screening data: 5% default rate
X = np.random.randn(2000, 10)
y = np.random.binomial(1, 0.05, 2000)

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(
        class_weight='balanced',
        solver='lbfgs',
        max_iter=1000
    ))
])

params = {'clf__C': [0.01, 0.1, 1, 10, 100]}
grid = GridSearchCV(pipeline, params, cv=5, scoring='roc_auc')
grid.fit(X, y)

print(f"Best C: {grid.best_params_['clf__C']}")
print(f"Best AUC: {grid.best_score_:.3f}")

Disadvantages of Logistic Regression — The 3 Hard Walls You'll Hit

Logistic regression is your starting gun, not your finishing line. It fails hard in three common production scenarios, and pretending otherwise costs you.

First, linear decision boundary. Logistic regression draws a straight line (or hyperplane) through feature space. If your data has XOR patterns — think credit risk where being both high-income AND high-debt is dangerous but either alone is safe — you need polynomial features, decision trees, or neural nets. You can engineer interactions manually, but that's guessing, not learning.

Second, multicollinearity kills coefficient interpretability. When two features are highly correlated (e.g., income and credit score), the model can't tell which one matters. Coefficients explode in opposite directions, making feature importance analysis useless. Senior engineers run variance inflation factor (VIF) checks before trusting coefficients.

Third, logistic regression can't learn complex feature interactions natively. If the signal lives in combinations of three or more features (e.g., age income geography), you need manual feature crossing or a model that builds hierarchies. XGBoost or a shallow neural net will crush LR on these problems. Know when to walk away from the tried-and-true.

Bottom line: LR is a fast, interpretable baseline. If you need non-linear boundaries, interaction learning, or robustness to collinearity, swap it out before your stakeholders ask why the model is stupid.

// io.thecodeforge — ml-ai tutorial

import pandas as pd
import numpy as np
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

# Simulated correlated features
np.random.seed(42)
data = pd.DataFrame({
    'income': np.random.normal(50000, 15000, 500),
    'credit_score': np.random.normal(700, 50, 500),
    'debt': np.random.normal(10000, 5000, 500)
})

# Artificially correlate credit_score with income
data['credit_score'] = data['income'] * 0.01 + np.random.normal(0, 10, 500)

X = add_constant(data)
vif = pd.DataFrame()
vif['feature'] = X.columns
vif['VIF'] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

print(vif)
# Features with VIF > 10 are dangerously collinear

Ordinal Logistic Regression — When Your Target Has a Natural Order

Standard logistic regression expects a binary outcome. Ordinal logistic regression extends this to categorical targets with an inherent ranking — like education level (high school, bachelor, master) or survey responses (poor, fair, good, excellent). The model assumes proportional odds: the effect of a feature is constant across all thresholds between categories. For example, the coefficient for 'years of experience' shifts the log-odds of moving from any lower category to any higher one by the same amount. This assumption must be verified via a Brant test or likelihood-ratio comparison against a model that relaxes it (e.g., multinomial). Fit using mord or statsmodels.miscmodels.ordinal_model. Output is a set of intercepts (thresholds) plus one shared coefficient vector. Predict class probabilities across all levels. Violating the proportional odds assumption biases coefficients and misranks predictions. Always test it — most practitioners miss this and get misleading feature importances.

// io.thecodeforge — ml-ai tutorial

import pandas as pd
import statsmodels.api as sm
from statsmodels.miscmodels.ordinal_model import OrderedModel

# Sample data: education (0=HS, 1=Bachelor, 2=Master), income in $k
df = pd.DataFrame({
    'edu': [0, 0, 1, 1, 2, 2],
    'income': [30, 40, 50, 60, 70, 80]
})

# Fit ordinal logistic regression
model = OrderedModel(df['edu'], df[['income']], distr='logit')
result = model.fit(method='bfgs')
print(result.summary())

Multinomial Logistic Regression — Why Softmax Replaces Sigmoid for Multi-Class

When you have more than two unordered classes — like classifying iris species — binary logistic regression fails. Multinomial logistic regression uses the softmax function to estimate probabilities across K categories: each outcome gets its own coefficient vector, and softmax normalizes so probabilities sum to 1. The model is trained using maximum likelihood with cross-entropy loss. Scikit-learn's LogisticRegression(multi_class='multinomial', solver='lbfgs') handles this directly. The reference category matters: coefficients are interpreted as log-odds relative to the baseline class (typically the first). Regularization still applies — use L2 to avoid overfitting with many categories. A critical drawback: the number of parameters grows linearly with K, requiring more data. For high-dimensional problems (e.g., text classification with 1000 classes), consider alternatives like naive Bayes or hierarchical softmax. Never use multinomial when classes are ordinal — you'd waste information about natural ordering.

// io.thecodeforge — ml-ai tutorial

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

iris = load_iris()
X, y = iris.data, iris.target

# Multinomial logistic regression
model = LogisticRegression(multi_class='multinomial', solver='lbfgs', max_iter=200)
model.fit(X, y)

# Predict probabilities for first sample
print('Probabilities:', model.predict_proba(X[:1]))
print('Predicted class:', model.predict(X[:1]))