Intermediate 5 min · March 06, 2026

Data Preprocessing in ML — Stopping Silent Data Leakage

A credit model's 0.

Naren · Founder

Plain-English first. Then code. Then the interview question.

⚡Quick Answer

Data preprocessing transforms raw, messy data into a clean format ML models can learn from
Handles missing values via imputation (median, most_frequent) plus missing indicator columns
Encodes categorical features: OneHotEncoder for nominal, OrdinalEncoder for ordinal
Scales numerical features to prevent large-valued features from dominating distance metrics
Splitting train/test BEFORE any fitting prevents data leakage — the #1 production bug
Use scikit-learn Pipeline + ColumnTransformer to chain steps safely and reproducibly

Every ML tutorial starts with a clean, perfectly formatted dataset. Real life never does. In the real world, data comes from messy CSV exports, broken sensors, rushed data-entry clerks, and legacy databases that mix text and numbers in the same column. The gap between raw data and model-ready data is where most ML projects actually live — and die. Skipping preprocessing is the single biggest reason a model that looked great in a notebook performs terribly in production.

Preprocessing solves three fundamental problems: data your model can't read (wrong types, text categories), data your model misreads (wildly different scales that trick distance-based algorithms), and data that simply isn't there (missing values that silently corrupt your results). Each of these problems has a well-understood solution, but the order and method you choose matter enormously depending on your data and your model.

By the end of this article you'll be able to audit a raw dataset, choose the right strategy for missing values, encode categorical features correctly, scale numerical features without leaking information from your test set, and wire everything together in a reproducible scikit-learn Pipeline. You'll also know the three mistakes that trip up intermediate practitioners — not just beginners.

Handling Missing Values — Why 'Just Drop Them' Is Usually Wrong

Missing data isn't random noise you can ignore. It's a signal. A missing income field in a loan application might mean the applicant refused to share it — which is itself predictive. Blindly dropping rows throws away that signal and shrinks your training set.

There are three strategies: deletion, imputation, and indicator flags. Deletion (dropping rows or columns) only makes sense when less than 5% of a column is missing AND missingness is truly random. Imputation replaces missing values with something plausible — the mean or median for numerical data, the most frequent value for categorical data, or a model-predicted value for high-stakes features.

The best practice for production is to combine imputation with a binary indicator column: a new column that says 'this value was missing' lets the model learn from the missingness pattern itself. Scikit-learn's SimpleImputer handles the replacement; you add the flag column manually before imputing.

Crucially, you must fit your imputer on training data only, then transform both train and test. Fitting on the full dataset leaks future information into your model — a subtle bug that inflates validation scores.

Encoding Categorical Features — Choosing Between Label, Ordinal, and One-Hot

Machine learning models are fundamentally mathematical. They multiply, add, and compare numbers. When your data has a column called 'City' with values like 'London', 'Paris', 'Tokyo', the model can't do anything with strings — you have to convert them.

The wrong choice here actively hurts your model. Label encoding assigns integers arbitrarily: London=0, Paris=1, Tokyo=2. That implies Tokyo > Paris > London mathematically, which is nonsense. Any model using arithmetic on those integers — linear regression, neural nets, SVMs — will learn a false relationship.

One-Hot Encoding (OHE) is the correct fix for nominal categories (no natural order). It creates a new binary column per category: is_London, is_Paris, is_Tokyo. No false ordering. The trade-off is that high-cardinality columns (e.g. 500 cities) explode your feature space — in that case, target encoding or embedding layers are better alternatives.

Ordinal encoding IS appropriate when the order genuinely matters: ['cold', 'warm', 'hot'] → [0, 1, 2] is correct because hot > warm > cold is real. Use OrdinalEncoder for these, not LabelEncoder (which is meant for target labels only).

Always handle unseen categories in your test set. A category that appears in production but wasn't in training will crash a naive encoder.

Feature Scaling — Why Your Algorithm's Math Demands It

Picture two features: age (18–65) and annual salary (30,000–150,000). The salary values are 3,000x larger. Any algorithm that computes distances or uses gradient descent treats the salary as 3,000x more important — purely because of measurement units, not because it actually matters more.

This kills k-Nearest Neighbours (distances dominated by salary), SVMs, and gradient descent convergence in neural nets. Tree-based models like Random Forest and XGBoost are the exception — they split on thresholds and don't care about absolute scale.

Two scalers solve this in different ways. StandardScaler subtracts the mean and divides by standard deviation, producing a distribution centred at 0 with unit variance. Use it when your data is roughly Gaussian or when the algorithm assumes it (linear/logistic regression, PCA, SVMs).

MinMaxScaler compresses values into a fixed range, typically [0, 1]. Use it when you need bounded outputs — for example, feeding pixel values into a neural network, or when the algorithm explicitly requires [0,1] input. Its weakness: a single extreme outlier squashes all other values into a tiny range.

RobustScaler uses the median and interquartile range instead of mean and standard deviation. It's your best friend when data has significant outliers — a faulty sensor reading of 999999 won't ruin your entire scaling.

Wiring It All Together With a scikit-learn Pipeline

You've now got individual tools for missing values, encoding, and scaling. The temptation is to apply them manually one by one in a sequence of function calls. Don't. Manual preprocessing has two fatal flaws: you'll inevitably leak training statistics into your test set (because it's easy to forget to split first), and you can't reliably reproduce or deploy the same sequence.

Scikit-learn's Pipeline chains transformers and a final estimator into a single object. When you call pipeline.fit(X_train, y_train), every transformer is fit on training data only and then applied in sequence. When you call pipeline.predict(X_test), transformers are applied using the already-fitted parameters — no leakage possible.

ColumnTransformer lets you apply different preprocessing to different columns inside the same Pipeline step. Numeric columns get imputed then scaled; categorical columns get imputed then one-hot encoded. Everything stays in sync.

This pattern also makes deployment trivial. You save one pipeline object with joblib. You load it in production. You call predict on raw, unprocessed input. The pipeline handles everything. No separate preprocessing script to maintain.

Outlier Detection and Treatment: When to Remove, Cap, or Transform

Outliers are data points that differ significantly from the rest. They can be genuine extreme values (e.g., a billionaire's income in a loan dataset) or errors (a sensor reading of 999.9°C). How you treat them depends on which case you're dealing with.

First, detect outliers. Common methods: Z-score (assumes normal distribution), IQR (robust, uses Q1-1.5IQR and Q3+1.5IQR), and domain-specific thresholds. For production, a combination works best: flag statistical outliers AND apply business rules (e.g., 'salary > $10M is impossible for our user base').

Once detected, you have three options. Remove: only when you're certain it's an error and you have enough data left. Cap (winsorize): replace outliers with the nearest non-outlier boundary — keeps the point but limits its influence. Transform: apply log or Box-Cox to reduce skew — makes the distribution more Gaussian and reduces outlier impact.

Never remove outliers blindly without understanding their origin. An outlier might be the most important data point — a fraud detection model must learn from extreme transaction amounts, not discard them.

Scaler Comparison at a Glance

Aspect	StandardScaler	MinMaxScaler	RobustScaler
Formula	(x - mean) / std	(x - min) / (max - min)	(x - median) / IQR
Output range	Unbounded (~-3 to 3)	Exactly [0, 1]	Unbounded, centred on median
Outlier sensitivity	High — outliers shift mean and std	Very high — single outlier dominates range	Low — uses median and IQR, ignores tails
Best for	Gaussian data, PCA, linear models, SVMs	Neural nets needing bounded input, image pixels	Data with known outliers (sensors, finance)
Loses interpretability?	Yes — values no longer in original units	Partially — proportional but shifted	Yes — relative to median not mean

Key Takeaways

Split your data BEFORE fitting any preprocessor — fitting on the full dataset leaks test-set statistics into training, silently inflating your validation scores.
Use OneHotEncoder for nominal categories (no natural order), OrdinalEncoder with explicit ordering for ordinal categories, and never LabelEncoder on input features.
RobustScaler is your go-to when data contains outliers — it uses median and IQR instead of mean and std, so a single bad sensor reading won't crush your entire feature range.
A scikit-learn Pipeline with ColumnTransformer isn't just convenience — it's the only production-safe way to guarantee that preprocessing is applied identically at train time and predict time without manual error.
Outlier treatment depends on origin: measurement errors should be removed or capped; extreme truths should be transformed (log) or kept with robust scaling.
Tree-based models don't need scaling; distance-based models and neural networks require it.

Common Mistakes to Avoid

Fitting preprocessors on the full dataset before train/test split
Symptom: Your imputer or scaler learns statistics from test data (e.g. the test set's median salary), which then influences training. Validation accuracy looks great but production performance is lower than expected.
Fix: Always call train_test_split first, then fit any preprocessor exclusively on X_train.
Using LabelEncoder on input features instead of OrdinalEncoder
Symptom: LabelEncoder encodes alphabetically (Berlin=0, London=1, Paris=2, Tokyo=3), implying Tokyo is mathematically 'greater than' London. Linear models and SVMs learn this false relationship and produce nonsense coefficients.
Fix: Use OneHotEncoder for nominal categories and OrdinalEncoder (with explicit category ordering) for ordinal ones. Reserve LabelEncoder for the target label only.
Not handling unseen categories in production
Symptom: A city or product type that never appeared during training causes OneHotEncoder to raise a ValueError at prediction time, crashing your API.
Fix: Always set handle_unknown='ignore' in OneHotEncoder when building production pipelines. The encoder will output an all-zero row for unknown categories instead of throwing an exception.
Applying scaling to tree-based models
Symptom: Unnecessary scaling adds compute cost without any benefit. Worse, if you later switch to a distance-based model, you might forget to un-scale, leading to confusion.
Fix: Only scale for k-NN, SVM, linear models, neural networks. Skip scaling for Random Forest, XGBoost, LightGBM.
Imputing missing values with the mean without checking for outliers
Symptom: If the column has extreme outliers, the mean is pulled away from the central tendency. Imputed values become unrealistic, distorting the distribution.
Fix: Use median imputation for numerical columns that may have outliers. Or apply a robust scaler after imputation.

Interview Questions on This Topic

QWhy must you fit your preprocessing transformers only on training data? What specifically goes wrong if you fit on the full dataset?Mid-levelReveal
Fitting on the full dataset leaks information from the test set into the training set. For example, if you compute the median salary on all data including test, then impute missing training salaries with that median, your model has effectively 'seen' test data. This artificially inflates validation scores — the model looks great offline but fails in production because test-set statistics won't be available at inference time. Always split first, then fit preprocessors on X_train only.
QWhen would you choose RobustScaler over StandardScaler, and can you give a concrete industry example where the difference matters?SeniorReveal
Choose RobustScaler when your data contains outliers that you don't want to remove but also don't want to dominate the scale. For example, in sensor fault detection, temperature readings occasionally spike to 999.9°C due to a hardware glitch. With StandardScaler, the spike shifts the mean and inflates the standard deviation, compressing normal readings into a narrow range. The model can't distinguish normal from not-normal. RobustScaler uses median and IQR, so the spike stays isolated and normal readings maintain their spread.
QWhy do tree-based models like Random Forest not require feature scaling, while logistic regression and SVMs do? What property of these algorithms explains this?Mid-levelReveal
Tree-based models make decisions by splitting on feature thresholds — they compare a value to a threshold, not compute distances across dimensions. So the absolute scale of each feature is irrelevant; only the ordering matters. Logistic regression and SVMs use gradient descent to optimise coefficients, and they rely on distance metrics (e.g. dot product, Euclidean distance). If one feature's values are 1000x larger than another's, its coefficient updates will dominate the gradient, and the smaller feature's influence vanishes. Scaling ensures all features contribute equally to the optimisation.
QExplain how you would handle a categorical feature with 500 unique categories in a logistic regression model without causing a dimensionality explosion.SeniorReveal
One-hot encoding 500 categories creates 500 binary columns, which is undeniably large but often manageable with modern compute if you have enough data. However, you can reduce dimensionality: apply target encoding (replace category with mean target value for that category, possibly with smoothing) or use embedding layers (for neural nets). For linear models, target encoding is effective but requires careful cross-validation to prevent target leakage. Another approach is to group rare categories into an 'other' bucket before encoding.
QWhat's the difference between imputing missing values and just dropping rows with NaN? When would you choose one over the other?JuniorReveal
Dropping rows is fast but wastes data and introduces bias if missingness is not completely random. For example, if people with low income consistently skip the income field, dropping them removes a pattern your model should learn. Imputation preserves all rows and, when combined with a 'was_missing' indicator, lets the model learn from absence itself. Drop only when <5% of rows are affected and you're confident missingness is random. Impute (with median/mode + indicator) for all other cases.

Frequently Asked Questions

What is the correct order for data preprocessing steps in machine learning?

Split into train and test first, then in this order: handle missing values (imputation), encode categorical features, scale numerical features, and finally feed into your model. Splitting first is non-negotiable — every other step must be fit on training data only. Wrapping all steps in a scikit-learn Pipeline enforces this order automatically.

Do I always need to scale features for every machine learning model?

No. Tree-based models like Decision Trees, Random Forests, and XGBoost are scale-invariant because they split on value thresholds — the absolute magnitude of a feature doesn't affect which split is chosen. Scaling is essential for distance-based algorithms (k-NN, SVM), linear models (logistic/linear regression), and neural networks, where large-valued features dominate gradient updates or distance calculations.

What's the difference between imputing missing values and just dropping rows with NaN?

Dropping rows is fast but wastes data and introduces bias if missingness isn't random — for example, if low-income people consistently skip the income field, dropping them removes a real pattern your model should learn. Imputation preserves all rows and, when combined with a 'was_missing' indicator column, actually lets the model learn from the fact that data was absent. Use deletion only when less than ~5% of a column is missing and you're confident the missingness is completely random.

How do I handle a categorical feature with hundreds of unique values without exploding my feature space?

One-hot encoding 500 categories creates 500 columns, which is often manageable with modern compute if you have enough data. To reduce dimensionality: use target encoding (replace each category with the mean target for that category, applied with cross-validation to avoid leakage), group rare categories into an 'Other' bucket, or use feature hashing. For neural networks, embedding layers are ideal. In scikit-learn, you can combine thresholding with OneHotEncoder: set min_frequency=0.05 to group rare categories.

What is data leakage and how do I prevent it in preprocessing?

Data leakage occurs when information from outside the training set influences the model — most commonly when preprocessors (imputers, scalers, encoders) are fit on the entire dataset before splitting. The fix: always call train_test_split first, then fit preprocessors only on X_train. Using Pipeline ensures this separation. Another form of leakage is target encoding using the full dataset — apply target encoding only within cross-validation folds.

Should I remove outliers before or after scaling?

Outlier detection and treatment should happen before scaling, because scaling can hide outliers (especially with StandardScaler where they skew the mean). Detect outliers on raw data, then decide: remove if you're sure it's an error; cap if it's extreme but real; then scale using RobustScaler if outliers remain. If you cap, do so before scaling so the boundaries are based on raw values.

What is the difference between LabelEncoder and OrdinalEncoder?

LabelEncoder is designed for encoding the target variable (y) — it transforms a single column of labels to integers 0..n_classes-1. OrdinalEncoder is for input features (X) and can handle multiple columns with explicit ordering. Using LabelEncoder on input features is a common mistake because it alphabetically orders categories, implying a false ordinal relationship. Use LabelEncoder only for the target, OrdinalEncoder (with categories parameter) for ordinal features.

Can I use StandardScaler on data that is not normally distributed?

Yes, but it may not produce good results. StandardScaler assumes the data is roughly bell-shaped. If your data is highly skewed, StandardScaler will still center and scale it, but the resulting distribution will still be skewed and outliers can distort the mean and standard deviation. For skewed data, apply a log or Box-Cox transformation first, then scale. Or use RobustScaler which is less sensitive to distribution shape.

How do I ensure my preprocessing is reproducible across environments?

Use a scikit-learn Pipeline serialized with joblib or pickle. Never replicate preprocessing steps manually in a new script. Save the fitted pipeline after training: joblib.dump(pipeline, 'model_pipeline.pkl'). In production, load it back: pipeline = joblib.load('model_pipeline.pkl'). This guarantees identical transformations. Also pin library versions (scikit-learn, pandas, numpy) in your deployment environment.

What is the difference between fit, transform, and fit_transform in scikit-learn?

fit() learns the parameters from the data (e.g., mean and std for StandardScaler). transform() applies the learned transformation to new data (e.g., standardizes values using the learned mean/std). fit_transform() is a convenience method that calls fit() then transform() on the same data — used on training data. Never call fit() on test data; only transform() using the parameters learned from training.

🔥

That's ML Basics. Mark it forged?

5 min read · try the examples if you haven't