Scikit-Learn Classification — OneHotEncoder Schema Drift

Classification predicts discrete class labels from labelled training data. Scikit-Learn provides a consistent, composable API for dozens of classification algorithms — from logistic regression to random forests — so you can swap algorithms, build preprocessing pipelines, evaluate performance rigorously, and tune hyperparameters without rewriting your code.

The algorithms are the easy part. The hard part is preventing data leakage, handling imbalanced classes, choosing the right evaluation metric, and building a pipeline you can serialize and deploy without surprises. This guide covers all of it — the algorithms, the gotchas, and the production patterns that separate a Jupyter notebook prototype from a reliable production system.

Here's the thing: most classification failures aren't about picking the wrong algorithm. They're about leaking test data into training, using accuracy on imbalanced data, or deploying a model that can't handle new categories. Get those right, and the algorithm choice often becomes a second-order concern.

What is Classification in Machine Learning?

Classification is a supervised learning task where the goal is to predict a discrete class label for each input. Supervised means you train on labelled examples — (features, label) pairs — where the correct label is known. The model learns the relationship between features and labels, then generalises to new inputs.

Binary classification has two classes (spam/not spam, fraud/not fraud, disease/healthy). Multi-class classification has three or more exclusive classes (cat, dog, bird). Multi-label classification assigns multiple labels per example (a news article can be both 'finance' and 'politics').

The output of a classifier is either a predicted class label (via predict()) or a probability distribution over all classes (via predict_proba()). Classification is distinct from regression, where the output is a continuous number.

The first question to ask before building any classifier: What does it cost to be wrong? If you misclassify spam, the user sees one extra email. If you misclassify a healthy patient as having cancer, they undergo unnecessary treatment. The cost of false positives vs false negatives determines your evaluation metric, your decision threshold, and your entire model selection strategy. Don't start with accuracy — start with the business impact of errors.

Here's a rule I've learned the hard way: if you don't know the cost of errors, you'll pick the wrong metric. And picking the wrong metric means you'll optimise the wrong thing. That's how you end up with a 99% accurate fraud model that catches zero fraud.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred, target_names=['setosa', 'versicolor', 'virginica']))

Scikit-Learn's Unified Estimator API — fit, predict, score

Scikit-Learn's biggest strength is its consistent API. Every classifier implements the same interface: fit(X, y) trains the model, predict(X) returns predicted labels, predict_proba(X) returns probability estimates, and score(X, y) returns mean accuracy.

This uniformity means you can swap algorithms with a single line change. The exact same preprocessing, splitting, and evaluation code works with LogisticRegression, RandomForestClassifier, SVC, or GradientBoostingClassifier.

The score() trap: score() returns accuracy by default for classifiers. For imbalanced datasets, accuracy is misleading — a model predicting the majority class every time scores 95% accuracy while being completely useless. Always use explicit metrics (F1, AUC) via sklearn.metrics rather than relying on score().

Think of it like this: score() is a shortcut for quick checks during development. But if you're using it to evaluate a model for production, you're making a mistake. You wouldn't check if a car works by looking at the colour — same idea.

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC

classifiers = {
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'Decision Tree':       DecisionTreeClassifier(max_depth=5),
    'Random Forest':       RandomForestClassifier(n_estimators=100),
    'Gradient Boosting':   GradientBoostingClassifier(n_estimators=100),
    'SVM':                 SVC(probability=True),
}

for name, clf in classifiers.items():
    clf.fit(X_train, y_train)
    acc = clf.score(X_test, y_test)
    print(f'{name}: {acc:.4f}')

probs = RandomForestClassifier().fit(X_train, y_train).predict_proba(X_test)
print(probs[0])

Feature Scaling — When to Scale and When Not to Bother

Feature scaling normalises the range of input features. Some algorithms are sensitive to feature scale; others are completely invariant.

Algorithms that REQUIRE scaling: SVM (distance-based), KNN (distance-based), Logistic Regression (gradient descent convergence), Neural Networks (gradient-based optimization). If you forget to scale for these, the feature with the largest range dominates the model.

Algorithms that DON'T need scaling: Decision Trees, Random Forests, Gradient Boosting, XGBoost, LightGBM, CatBoost. Tree-based models split on individual feature thresholds, so absolute scale doesn't matter.

Three scalers to know: StandardScaler (zero mean, unit variance — sensitive to outliers), MinMaxScaler (scales to [0,1] — for neural networks), RobustScaler (median and IQR — robust to outliers).

The production rule: If your pipeline includes SVM, KNN, or Logistic Regression, add StandardScaler. If it's tree-based only, skip scaling. If unsure, add it — it won't hurt tree models, just wastes a few milliseconds.

One more thing: don't just blindly scale everything. I've debugged cases where scaling a sparse binary feature caused the model to behave poorly. Know your data.

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.neural_network import MLPClassifier

# SVM NEEDS scaling
svm_scaled = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', SVC(kernel='rbf', probability=True)),
])

# When data has outliers, use RobustScaler
svm_robust = Pipeline([
    ('scaler', RobustScaler()),
    ('clf', SVC(kernel='rbf', probability=True)),
])

# For neural networks or bounded-input models
nn_scaled = Pipeline([
    ('scaler', MinMaxScaler()),
    ('clf', MLPClassifier(hidden_layer_sizes=(100, 50), max_iter=500)),
])

Preprocessing Pipelines — The Right Way to Handle Feature Engineering

A pipeline chains preprocessing steps and a classifier into a single object. This is not optional convenience — it is the correct way to prevent data leakage.

Data leakage happens when information from the test set influences training. The classic mistake: fit a StandardScaler on the entire dataset before splitting. The scaler has 'seen' the test data and computed statistics from it. Your model was trained on a subtly contaminated version of reality.

With a Pipeline, fit() calls fit_transform() on preprocessors and fit() on the classifier — all on training data only. predict() calls transform() on preprocessors and predict() on the classifier. The test data is only transformed with statistics learned from training data.

The ColumnTransformer pattern: Real datasets have mixed feature types — numeric columns (age, income), categorical columns (country, plan_type). ColumnTransformer applies different preprocessing to different column subsets, all within the same pipeline. This is the standard pattern for production ML.

I'll say it again: if you're not using a Pipeline, you're almost certainly leaking data. I've seen it countless times.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
import numpy as np
import pandas as pd

# Simulate a realistic dataset
df = pd.DataFrame({
    'age': [25, 45, np.nan, 30, 60],
    'income': [50000, 80000, 120000, np.nan, 90000],
    'country': ['US', 'UK', 'US', 'DE', 'FR'],
    'plan_type': ['basic', 'premium', 'basic', 'enterprise', 'premium'],
    'education': ['high_school', 'bachelors', 'masters', 'phd', 'bachelors'],
    'churned': [0, 0, 1, 0, 1],
})

X = df.drop('churned', axis=1)
y = df['churned']

numeric_features = ['age', 'income']
nominal_features = ['country', 'plan_type']
ordinal_features = ['education']

# Ordinal encoding for ordered categories
education_order = ['high_school', 'bachelors', 'masters', 'phd']

preprocessor = ColumnTransformer([
    ('num', Pipeline([('imputer', SimpleImputer(strategy='median')),
                      ('scaler', StandardScaler())]), numeric_features),
    ('nom', Pipeline([('imputer', SimpleImputer(strategy='most_frequent')),
                      ('onehot', OneHotEncoder(handle_unknown='ignore'))]), nominal_features),
    ('ord', Pipeline([('imputer', SimpleImputer(strategy='most_frequent')),
                      ('ordinal', OrdinalEncoder(categories=[education_order]))]), ordinal_features),
])

pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42)),
])

pipeline.fit(X, y)

# See transformed feature names
feature_names = pipeline.named_steps['preprocessor'].get_feature_names_out()
print('Transformed features:', feature_names)

Naive Bayes — The Baseline You Should Always Try First

Before reaching for Random Forest or XGBoost, try Naive Bayes. It's fast, simple, surprisingly effective, and serves as a strong baseline. If your complex model can't beat Naive Bayes, something is wrong with your features.

Three variants: GaussianNB (continuous features, assumes normal distribution), MultinomialNB (discrete count features like word counts or TF-IDF — the go-to for text classification), BernoulliNB (binary features — word present/absent).

Why it works for text: Text data is high-dimensional and sparse. Naive Bayes handles this gracefully because it assumes feature independence. This assumption is obviously wrong (words aren't independent), but it works shockingly well in practice.

The production baseline pattern: Always train a Naive Bayes model first. Report its metrics. Then train your fancy model. If the fancy model only marginally beats Naive Bayes, consider whether the added complexity is worth it. Naive Bayes trains in milliseconds and predicts in microseconds — that matters for real-time systems.

Here's a story: I once replaced a carefully tuned XGBoost model with a Naive Bayes model for a real-time ad classification system. The XGBoost was 1.2% more accurate but took 15x longer to predict. The business chose the faster model. Know your constraints.

from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

# Gaussian Naive Bayes for continuous features
gnb = GaussianNB()
gnb.fit(X_train, y_train)
y_pred = gnb.predict(X_test)
print('GaussianNB:', classification_report(y_test, y_pred))

# Multinomial Naive Bayes for text (TF-IDF features)
text_pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=5000, stop_words='english')),
    ('clf',   MultinomialNB(alpha=0.1)),
])

# Bernoulli Naive Bayes for binary features
bnb = BernoulliNB()
binary_features = (X_train > 0).astype(int)
bnb.fit(binary_features, y_train)

Evaluating a Classifier — Beyond Accuracy

Accuracy is misleading for imbalanced classes. If 95% of data is class 0, a classifier predicting class 0 always achieves 95% accuracy while being completely useless.

Precision: Of all samples predicted positive, what fraction actually are? High precision = few false alarms. Recall: Of all actual positives, what fraction did we catch? High recall = few missed cases. F1: Harmonic mean of precision and recall.

The confusion matrix shows the full breakdown: TP, TN, FP, FN. For multi-class, it's an NxN matrix where the diagonal is correct predictions.

ROC-AUC measures discrimination ability across all thresholds. AUC 0.5 = random, 1.0 = perfect. PR-AUC (Precision-Recall AUC) is better for imbalanced data because it focuses on the positive class.

Which metric matters depends on the business cost: - Spam filtering: Precision matters. You don't want legitimate email in the spam folder. - Disease screening: Recall matters. You don't want to miss a sick patient. - Fraud detection: Recall matters more, but precision also matters because investigating false positives costs money.

The multi-metric approach: Always report at least three metrics: precision, recall, and F1. Add AUC if you use predicted probabilities. Never report accuracy alone for imbalanced problems.

I've had to tell more than one team that their 99% accurate model was useless. The confusion matrix showed they predicted 'not fraud' for everything. That's when you know you've been optimising the wrong thing.

from sklearn.metrics import (
    classification_report, confusion_matrix,
    roc_auc_score, average_precision_score,
    ConfusionMatrixDisplay, precision_recall_curve
)
import matplotlib.pyplot as plt
import numpy as np

print(classification_report(y_test, y_pred))

cm = confusion_matrix(y_test, y_pred)
disp = ConfusionMatrixDisplay(confusion_matrix=cm)
disp.plot()
plt.title('Confusion Matrix')
plt.show()

# ROC-AUC for binary classification
probs = pipeline.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, probs)
print(f'ROC-AUC: {auc:.4f}')

# PR-AUC — better metric for imbalanced classes
pr_auc = average_precision_score(y_test, probs)
print(f'PR-AUC:  {pr_auc:.4f}')

# Custom class weights — when you know the business cost ratio
clf_cost_sensitive = RandomForestClassifier(
    class_weight={0: 1, 1: 100},
    random_state=42
)
clf_cost_sensitive.fit(X_train, y_train)
y_pred_cost = clf_cost_sensitive.predict(X_test)
print(classification_report(y_test, y_pred_cost))

Decision Threshold Tuning — The Most Underrated Technique

Every binary classifier has a default decision threshold of 0.5: if predict_proba() returns >= 0.5, predict class 1. This threshold is arbitrary and almost never optimal for your specific business problem.

The insight: The threshold controls the trade-off between precision and recall. Lowering it catches more positives (higher recall) but flags more negatives as positives (lower precision). Raising it gives fewer but more trustworthy positive predictions.

How to find the optimal threshold: Use precision_recall_curve to get precision and recall at every possible threshold. Then choose the threshold that optimises your business metric.

In production: Don't use predict() — use predict_proba() and apply your own threshold. Store the threshold alongside the model. When business costs change, adjust the threshold without retraining.

I tuned the threshold on a fraud detection model from 0.5 to 0.15. Recall went from 60% to 92%. Precision dropped from 85% to 45%. The fraud team preferred catching 92% of fraud — the cost of missing fraud far exceeded the cost of investigating false positives. That's a business decision, not a technical one.

import numpy as np
from sklearn.metrics import precision_recall_curve, f1_score

# Get probabilities
y_probs = pipeline.predict_proba(X_test)[:, 1]
precisions, recalls, thresholds = precision_recall_curve(y_test, y_probs)

# Find threshold that maximises F1
f1_scores = 2 * precisions[:-1] * recalls[:-1] / (precisions[:-1] + recalls[:-1] + 1e-8)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]
print(f'Best threshold for F1: {best_threshold:.3f}')
print(f'F1 at best threshold: {f1_scores[best_idx]:.3f}')

# Apply custom threshold
y_pred_custom = (y_probs >= best_threshold).astype(int)
print(f'F1 with custom threshold: {f1_score(y_test, y_pred_custom):.4f}')

# Find threshold for minimum recall target
min_recall = 0.95
valid_indices = np.where(recalls[:-1] >= min_recall)[0]
if len(valid_indices) > 0:
    best_for_recall = valid_indices[np.argmax(precisions[valid_indices])]
    recall_threshold = thresholds[best_for_recall]
    print(f'Threshold for >= 95% recall: {recall_threshold:.3f}')

# Business cost minimisation
fn_cost = 1000  # missed fraud
fp_cost = 10    # false alarm
min_cost = float('inf')
best_cost_threshold = 0.5
for i, t in enumerate(thresholds):
    y_pred_t = (y_probs >= t).astype(int)
    fn = np.sum((y_test == 1) & (y_pred_t == 0))
    fp = np.sum((y_test == 0) & (y_pred_t == 1))
    cost = fn * fn_cost + fp * fp_cost
    if cost < min_cost:
        min_cost = cost
        best_cost_threshold = t
print(f'Threshold minimising business cost: {best_cost_threshold:.3f} (cost: ${min_cost:,.0f})')

Handling Imbalanced Classes — SMOTE, Thresholds, and Class Weights

Most real-world classification problems are imbalanced: fraud is rare, churn is rare, disease is rare. If you don't address this, your model will learn to predict the majority class every time.

1. class_weight='balanced': The simplest approach. Increases the loss contribution of minority class samples. Works with LogisticRegression, RandomForest, SVM. No extra dependencies. Try this first.

2. SMOTE (Synthetic Minority Oversampling): Generates synthetic minority class samples by interpolating between existing minority samples. pip install imbalanced-learn. Use SMOTEENN or SMOTETomek to clean noisy synthetic samples.

3. Threshold tuning: Use predict_proba() and tune the decision threshold (see previous section). Often the most effective approach because you keep the model unchanged — you just change the decision boundary.

The gotcha with SMOTE: Never apply SMOTE before train/test splitting. SMOTE generates synthetic samples based on nearest neighbours — if applied before splitting, synthetic test samples leak information about training samples. Always SMOTE inside a pipeline or after splitting.

imblearn.pipeline.Pipeline: scikit-learn's Pipeline doesn't support samplers (they lack transform()). Use imblearn.pipeline.Pipeline instead, which supports both transformers and samplers.

I've seen too many people apply SMOTE before splitting and claim an F1 of 0.95. When they fix it, it drops to 0.65. Don't be that person.

from imblearn.over_sampling import SMOTE, ADASYN
from imblearn.under_sampling import RandomUnderSampler
from imblearn.combine import SMOTEENN
from imblearn.pipeline import Pipeline as ImbPipeline
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np

# Strategy 1: class_weight (simplest)
clf_weighted = RandomForestClassifier(n_estimators=200, class_weight='balanced', random_state=42)

# Strategy 2: SMOTE inside imblearn pipeline
preprocessor = ColumnTransformer([
    ('num', Pipeline([('imp', SimpleImputer(strategy='median')), ('sc', StandardScaler())]), numeric_features),
    ('cat', Pipeline([('imp', SimpleImputer(strategy='most_frequent')), ('ohe', OneHotEncoder(handle_unknown='ignore'))]), categorical_features),
])

smote_pipeline = ImbPipeline([
    ('preprocessor', preprocessor),
    ('smote', SMOTE(random_state=42)),
    ('classifier', RandomForestClassifier(n_estimators=200, random_state=42)),
])

# Strategy 3: SMOTE + Edited Nearest Neighbours (cleans noisy samples)
smoteenn_pipeline = ImbPipeline([
    ('preprocessor', preprocessor),
    ('smoteenn', SMOTEENN(random_state=42)),
    ('classifier', RandomForestClassifier(n_estimators=200, random_state=42)),
])

# Strategy 4: Undersampling + SMOTE
combined_pipeline = ImbPipeline([
    ('preprocessor', preprocessor),
    ('under', RandomUnderSampler(sampling_strategy=0.5, random_state=42)),
    ('smote', SMOTE(random_state=42)),
    ('classifier', RandomForestClassifier(n_estimators=200, random_state=42)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for name, pipe in [('class_weight', clf_weighted),
                   ('SMOTE', smote_pipeline),
                   ('SMOTEENN', smoteenn_pipeline),
                   ('Under+SMOTE', combined_pipeline)]:
    scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring='f1')
    print(f'{name:15s} F1: {scores.mean():.4f} (+/- {scores.std():.4f})')

Cross-Validation — Reliable Performance Estimates

A single train/test split gives a noisy estimate. The specific random split affects which samples are in each set, and performance can vary significantly between splits. Cross-validation averages performance across multiple splits.

K-fold splits data into K folds, trains on K-1, validates on the remaining fold, rotates K times. StratifiedKFold preserves class proportions — always use it for classification.

cross_val_score returns test scores. cross_validate returns train and test scores plus timing — useful for diagnosing overfitting (train >> test = overfitting).

How many folds? 5 is the default. Use 10 for small datasets (more training data per fold). Use 3 for very large datasets (faster). The standard deviation across folds shows how stable the estimate is.

I once had a model that scored 0.95 on one random split and 0.72 on another. Cross-validation showed the true score was 0.83 with a standard deviation of 0.08. That's why we use CV.

from sklearn.model_selection import cross_val_score, cross_validate, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
clf = RandomForestClassifier(n_estimators=100, random_state=42)

scores = cross_val_score(clf, X, y, cv=cv, scoring='f1_weighted')
print(f'F1: {scores.mean():.4f} (+/- {scores.std():.4f})')

results = cross_validate(clf, X, y, cv=cv, scoring=['accuracy', 'f1_weighted'], return_train_score=True)
print('Train F1:', results['train_f1_weighted'].mean())
print('Test  F1:', results['test_f1_weighted'].mean())

# CV with full pipeline — no leakage
cv_scores = cross_val_score(pipeline, X, y, cv=cv, scoring='roc_auc')
print(f'Pipeline AUC: {cv_scores.mean():.4f}')

Logistic Regression — Not Actually Regression, and Why That Matters

Junior devs hear "regression" and think continuous outputs. Logistic regression is a misnomer that's stuck around. It's a linear classifier that squashes its output through a sigmoid function to spit out class probabilities, not numbers. The decision boundary is a hyperplane. That's it. No magic.

You train it by minimizing log-loss, which punishes confident wrong predictions harder than a linear loss would. That property alone makes logistic regression a decent baseline for binary problems, especially when you need to explain why a model flagged a transaction as fraudulent. Coefficients have natural interpretations — a one-unit increase in feature X multiplies the odds by exp(coef).

But here's the catch: it assumes features are independent of each other (no multicollinearity). If you dump 50 highly correlated features into it, the coefficients start oscillating like a bad amplifier. Standardize your features first, or L2 regularization will bail you out. Use LogisticRegression(penalty='l2', C=1.0) as your starting point. Drop C to increase regularization strength when you see coefficients blowing up.

Production reality: logistic regression is your fast, interpretable baseline. If a random forest beats it by 2% on a small dataset, you don't need the black box. If logistic regression is competitive, you save yourself a heap of debugging later.

// io.thecodeforge — ml-ai tutorial

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(penalty="l2", C=1.0, random_state=42))
])

# Dummy transaction data: [amount, hour, merchant_id_encoded]
X_train = [[120.5, 14, 3], [45.0, 22, 1], [890.0, 3, 7]]
y_train = [0, 0, 1]

scores = cross_val_score(pipeline, X_train, y_train, cv=3, scoring="f1")
print(f"Mean F1: {scores.mean():.3f} (std {scores.std():.3f})")

K-Nearest Neighbors — The Lazy Learner That Demands Respect

KNN doesn't train a model. It memorizes the training set and classifies new points by majority vote among the k closest neighbors. This is the laziest learning algorithm in scikit-learn, and it's also one of the easiest to mess up in production.

The decision boundary is non-linear and adapts to local structure, which sounds great until you realize the entire algorithm is basically a nearest-neighbour search at inference time. On a dataset with 100k rows and 50 features, every prediction means computing distances to every training point. That's O(nd) per query. Your API latency will tank.

The only hyperparameter that matters is k. Too small (k=1) and you overfit to noise — a single outlier votes. Too large (k=50) and you smooth out the real signal, especially near class boundaries. Rule of thumb: start with k = sqrt(n_samples) and tune from there. Weighted voting (weights='distance') often helps when classes overlap.

Feature scaling is mandatory. KNN uses Euclidean distance. If you have one feature in thousands (salary) and another in decimals (age), the salary dimension dominates. StandardScaler or MinMaxScaler — pick one and pipeline it before KNeighborsClassifier.

Production shortcut: use KNN only when your dataset is under 10k rows or you've precomputed a metric tree (ball_tree or kd_tree). For anything larger, switch to a linear model or a tree ensemble.

// io.thecodeforge — ml-ai tutorial

from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

# Customer segmentation: [age, annual_income_k, spending_score]
X_train = np.array([[25, 45, 78], [42, 120, 35], [31, 60, 92]])
y_train = np.array([0, 1, 0])

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("knn", KNeighborsClassifier(n_neighbors=3, weights="distance"))
])

pipeline.fit(X_train, y_train)
# New customer: age=28, income=55k, spending=85
new_customer = np.array([[28, 55, 85]])
pred = pipeline.predict(new_customer)
print(f"Segment: {pred[0]}")

Decision Trees — Why Your First Tree Probably Overfits

Decision trees are the most intuitive model in scikit-learn: a series of if-else questions on features, splitting data into purer subsets. They're also the easiest way to shoot yourself in the foot. Without constraints, a tree will grow until every leaf contains one sample, memorizing noise. That's a perfect training accuracy and a garbage test score.

You control overfitting with four parameters: max_depth, min_samples_split, min_samples_leaf, and max_features. Start with max_depth=5 and min_samples_leaf=10 and work backwards. A leaf with fewer than 5 samples is usually noise, not signal. Pruning post-training (cost complexity pruning via ccp_alpha) is more principled but rarely used because random forests handle the same problem better.

Trees are sensitive to data rotations. Rotate the feature space and the splits change entirely. That means they handle feature interactions naturally (no need to engineer interaction terms), but they struggle with linear relationships that require diagonal boundaries. If your data has clear linear separators, logistic regression will beat a single tree.

Production rule: a single decision tree is never your final model in a serious system. Use it as a weak learner in a gradient boosting ensemble or as a simple baseline for interpretability (max_depth=3). Export it as a diagram with export_graphviz to explain it to non-technical stakeholders. Anything deeper and you're better off with RandomForestClassifier.

// io.thecodeforge — ml-ai tutorial

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV

# Loan default features: [credit_score, income_k, loan_amount_k, debt_ratio]
X_train = [[650, 55, 20, 0.35], [720, 90, 30, 0.20], [580, 35, 15, 0.60]]
y_train = [0, 0, 1]

param_grid = {
    "max_depth": [3, 5, 7],
    "min_samples_leaf": [5, 10, 15]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid,
    cv=3,
    scoring="recall"
)
grid.fit(X_train, y_train)
print(f"Best params: {grid.best_params_}")
print(f"Best recall: {grid.best_score_:.3f}")

Step-by-Step Implementation: From Raw Data to Classified Output

Before writing a single line of code, you must understand why the implementation order matters. Machine learning pipelines fail not because the algorithm is wrong, but because the data preparation order is incorrect. The why is simple: classifiers expect numerical, scaled, and properly shaped input. Start by importing essential libraries: pandas for data loading, scikit-learn's train_test_split, StandardScaler, and your chosen classifier. Always split data before scaling to prevent data leakage from the test set into the training statistics. Next, instantiate a pipeline with preprocessing steps and the classifier. Fit the pipeline on training data, then predict on test data. Evaluate using a confusion matrix and classification report, not just accuracy. The key insight: implement in layers — load, split, transform, train, evaluate. Each step depends on the previous one. Skipping feature scaling for distance-based models like SVM or KNN guarantees poor performance. This structured flow turns a chaotic experiment into a reproducible process. Remember: your model is only as good as your data handling sequence.

// io.thecodeforge — ml-ai tutorial
// 25 lines max
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

df = pd.read_csv('data.csv')
X = df.drop('target', axis=1)
y = df['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', SVC(kernel='rbf', C=1.0, gamma='scale'))
])

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
print(classification_report(y_test, y_pred))

Unsupervised Learning in Python: Why Clustering Comes Before Classification

Unsupervised learning is not a separate discipline; it is the bedrock of understanding your data before labeling it. The why is pragmatic: most real-world data lacks labels, and even when labels exist, clustering reveals hidden subgroups that accuracy metrics miss. Start with K-Means for its speed and interpretability, but only after scaling features — distance metrics demand equal variance contributions. Use the elbow method or silhouette score to determine optimal cluster count. DBSCAN is essential for non-spherical clusters and outlier detection, as it does not force every point into a cluster. After clustering, assign pseudo-labels and treat them as a new target for a supervised classifier. This semi-supervised approach often outperforms pure supervised learning when labeled data is scarce. Apply PCA to visualize high-dimensional clusters in 2D. The implementation flow: scale, cluster, evaluate cohesion, assign labels, then train a classifier on those labels. Unsupervised learning is not the end goal — it is the exploratory phase that informs every subsequent modeling decision.

// io.thecodeforge — ml-ai tutorial
// 25 lines max
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

df = pd.read_csv('unlabeled_data.csv')
scaler = StandardScaler()
X_scaled = scaler.fit_transform(df)

kmeans = KMeans(n_clusters=3, random_state=42)
labels = kmeans.fit_predict(X_scaled)

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=labels, cmap='viridis')
plt.title('Cluster Visualization')
plt.show()