Feature Engineering and Preprocessing in Scikit-Learn

Feature Engineering and Preprocessing in Scikit-Learn is foundational to every ML project that ships to production. Raw data is almost never ready for a mathematical algorithm to consume directly. It arrives with missing values, categorical text strings, features measured on wildly different scales, outliers that distort learned parameters, and distributions that violate model assumptions.

Scikit-Learn was designed with a consistent solution to this: the Transformer interface. Every preprocessing step exposes the same two methods — fit() to learn parameters from data, and transform() to apply those learned parameters to any dataset. That consistency is not cosmetic. It is what makes preprocessing steps composable, testable, and safe to plug into cross-validation loops without leaking information across folds.

At TheCodeForge, we treat preprocessing as the primary driver of model accuracy — not an afterthought. A well-tuned model on poorly prepared data will consistently lose to a simpler model on well-prepared data. The ceiling of what your model can learn is set by the quality of your preprocessing decisions, not by your choice of algorithm.

By the end of this guide you will understand why the fit/transform separation exists, how to apply each technique to the right kind of data, how to build preprocessing into a Pipeline that is safe for cross-validation, and where production systems break when the preprocessing step is handled carelessly.

What Is Feature Engineering and Preprocessing in Scikit-Learn and Why Does It Exist?

Feature Engineering and Preprocessing exists in Scikit-Learn because machine learning algorithms are fundamentally mathematical — they operate on numbers, distances, gradients, and matrix operations. Human-readable data does not arrive in that form. It arrives as salary figures in the hundreds of thousands sitting alongside age values between 0 and 100, department names like 'Engineering' and 'Marketing', and missing values scattered throughout.

Without preprocessing, a distance-based model like KNN would treat a salary difference of 50,000 dollars as astronomically more significant than an age difference of 10 years — not because salary actually matters more, but because the numbers are larger. The model never gets a chance to discover the real signal because the units of measurement are drowning it out.

Scikit-Learn's answer to this is the Transformer interface: a consistent pattern where every preprocessing step implements fit() to learn parameters from data and transform() to apply those parameters to any dataset. That two-method contract is the entire foundation. Once you internalize it, every preprocessing class in the library — all fifty-plus of them — follows the same mental model.

The fit/transform separation is not just an API design choice. It is a correctness requirement. The parameters learned during fit() on training data must be reused when transforming test and production data. If you refit on each new dataset, you introduce distribution mismatch. If you fit on all data before splitting, you introduce leakage. The separation enforces the only correct behavior: learn once from training data, apply everywhere else.

ColumnTransformer extends this to real-world datasets where numerical columns need scaling, categorical columns need encoding, and text columns might need entirely different treatment — all applied in parallel and concatenated into a single output matrix that a model can consume.

import numpy as np
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# io.thecodeforge: Production-grade preprocessing with correct split ordering

# Sample dataset: [Age, Salary, Department, Target]
# In production this comes from a database query or Parquet file
X = np.array([
    [25, 50000, 'Engineering'],
    [30, 80000, 'Marketing'],
    [45, 120000, 'Engineering'],
    [28, 62000, 'Marketing'],
    [35, 95000, 'Engineering'],
], dtype=object)

y = np.array([0, 1, 0, 1, 0])  # binary target

# STEP 1: Split BEFORE any transformer sees the data.
# This is non-negotiable. Everything downstream of this line
# must only ever call fit() on X_train.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# STEP 2: Define transformers for each column type.
# Column indices here correspond to the columns in X:
#   [0] = Age (numerical), [1] = Salary (numerical), [2] = Department (categorical)
#
# RobustScaler is used instead of StandardScaler because salary
# data in real datasets almost always has outliers (executive compensation).
# Using mean/std on a feature with extreme outliers produces poor normalization.
numerical_transformer = StandardScaler()  # swap for RobustScaler if outliers exist
categorical_transformer = OneHotEncoder(handle_unknown='ignore', sparse_output=False)

# STEP 3: Combine transformers into a ColumnTransformer.
# Each tuple is: (name, transformer, column_indices).
# The remainder='passthrough' default is intentionally avoided here —
# every column should be explicitly assigned to prevent silent passthrough
# of raw, unscaled features into the model.
preprocessor = ColumnTransformer(
    transformers=[
        ('numerical', numerical_transformer, [0, 1]),
        ('categorical', categorical_transformer, [2])
    ],
    remainder='drop'  # explicit: columns not listed are dropped, not passed through
)

# STEP 4: Wrap preprocessor and model in a Pipeline.
# This is the safety net. Pipeline guarantees that during cross-validation,
# fit() is called only on training folds — test folds are never seen by fit().
# It also means the preprocessing and model travel together as a single artifact.
forge_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression(max_iter=1000, random_state=42))
])

# STEP 5: Fit the entire Pipeline on training data only.
# Internally: preprocessor.fit_transform(X_train) then classifier.fit(X_train_processed, y_train)
forge_pipeline.fit(X_train, y_train)

# STEP 6: Transform test data using training parameters — never refit.
# Internally: preprocessor.transform(X_test) then classifier.predict(X_test_processed)
test_score = forge_pipeline.score(X_test, y_test)

print(f"Processed training shape: {preprocessor.fit_transform(X_train).shape}")
print(f"Pipeline test accuracy: {test_score:.2f}")

Enterprise Data Cleansing: SQL Pre-Aggregation

In any production ML system of meaningful scale, preprocessing begins before Python ever sees the data. SQL is the right tool for extraction, filtering, joins, and deterministic feature creation — operations that do not depend on dataset statistics and therefore carry no leakage risk.

The distinction is important and worth being precise about. A log transformation of salary — log(salary + 1) — is deterministic. The result depends only on the individual row value, not on the distribution of the column across all rows. Computing it in SQL is safe. Imputing a missing salary with the column mean, however, requires knowing the mean — which means you need to decide: mean of what rows? If the answer is 'all rows including test rows,' you have leakage. SQL pre-aggregation does not know about your train/test split. Scikit-Learn's SimpleImputer inside a Pipeline does.

The practical division: use SQL to retrieve the data in the right shape, drop clearly invalid rows, apply deterministic mathematical transforms, and join feature tables together. Use Scikit-Learn for anything that computes a statistic across rows — imputation, scaling, encoding vocabularies — because those operations must be confined to training data.

For datasets in the tens of millions of rows, this division also matters for performance. A GROUP BY aggregation or a window function in SQL running on a warehouse with parallel execution is orders of magnitude faster than the equivalent pandas operation. Pulling raw rows into Python and aggregating there burns memory and time unnecessarily.

-- io.thecodeforge: Deterministic feature engineering in SQL before Python ingestion
-- SAFE to do in SQL: filtering, joins, deterministic transforms, null drops
-- NOT SAFE to do in SQL: mean/median imputation, percentile-based binning,
--   any transform that computes a statistic across all rows including test rows

SELECT
    user_id,

    -- Deterministic null handling: replace with a known constant, not a dataset statistic.
    -- Using AVG(age) across all rows here would leak test-set statistics into the feature.
    -- If age is null, we will handle imputation in Scikit-Learn SimpleImputer instead.
    age,  -- leave nulls for Scikit-Learn to impute on training data only

    -- Deterministic mathematical transform: log scale compresses right-skewed salary data.
    -- log(0) is undefined, so we add 1 before applying (standard convention: log1p).
    -- This is safe in SQL because it depends only on the individual row value.
    LOG(salary + 1) AS log_salary,

    -- Deterministic binary flag: depends only on a fixed date threshold, not on data statistics.
    -- This is a business rule, not a learned parameter — safe to compute in SQL.
    CASE
        WHEN signup_date > '2025-01-01' THEN 1
        ELSE 0
    END AS is_new_user,

    -- Deterministic ratio feature: depends only on values within the same row.
    -- Safe to compute in SQL.
    CASE
        WHEN years_employed > 0 THEN ROUND(salary / years_employed, 2)
        ELSE NULL  -- avoid division by zero; let Scikit-Learn handle the null
    END AS salary_per_year,

    -- Target label included for supervised learning
    churn_label

FROM io.thecodeforge.raw_user_data

-- Filter clearly invalid rows before they reach Python.
-- This is data quality, not statistical preprocessing — safe to do in SQL.
WHERE is_active = true
  AND salary > 0
  AND age BETWEEN 18 AND 100;

Standardizing Preprocessing with Docker

Dependency drift is one of the most underappreciated sources of ML production failures. Scikit-Learn's transformers are implemented in Python and C. Minor version changes — sometimes even patch releases — can alter the output of transformers like PolynomialFeatures, change the random state behavior of certain samplers, or introduce subtle numerical differences in floating-point computations. If your training environment runs scikit-learn==1.4.1 and your inference environment runs scikit-learn==1.5.0, the fitted Pipeline you serialized during training may produce numerically different results when loaded for inference.

This is not hypothetical. The scikit-learn changelog documents breaking changes in transformer output across minor versions, including changes to default parameter values that silently alter behavior for anyone relying on those defaults.

Docker solves this by making the environment an artifact of the project rather than a property of the machine. The same base image, the same pinned library versions, and the same system-level scientific computing libraries run identically on a developer's laptop, in CI, and in the production inference service. The container is the environment contract.

For ML specifically, this matters beyond just reproducibility. When a model's predictions change unexpectedly in production, you need to be able to rule out environment differences immediately. If training and inference run from the same Docker image built from the same Dockerfile, the environment is ruled out in seconds. That eliminates an entire debugging axis from your incident response.

# io.thecodeforge: Immutable Preprocessing and Inference Environment
# This Dockerfile is the environment contract for this ML project.
# Training and inference MUST use the same image tag.

FROM python:3.11-slim

WORKDIR /app

# Install system-level scientific computing dependencies.
# libatlas-base-dev provides BLAS/ATLAS for NumPy and SciPy linear algebra.
# gfortran is required by SciPy for certain compiled extensions.
# Pinning the apt packages is not practical (versions managed by Debian),
# so we pin at the Python layer instead.
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        libatlas-base-dev \
        gfortran \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements first to leverage Docker layer caching.
# If requirements.txt hasn't changed, this layer is served from cache
# and the pip install step is skipped on rebuild — saves 2-5 minutes.
COPY requirements.txt .

# Pin EXACT versions — not ranges, not minimums.
# scikit-learn==1.4.2 not scikit-learn>=1.4
# A minor version bump can silently alter transformer output.
# See: https://scikit-learn.org/stable/whats_new/
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# The preprocessing script loads the fitted Pipeline from joblib
# and applies it to new data. Both training and inference import
# from the same path — the Pipeline travels with the container.
CMD ["python", "ForgePreprocessing.py"]

Common Mistakes and How to Avoid Them

Most preprocessing failures in production trace back to one of four mistakes. They are remarkably consistent across teams, seniority levels, and problem domains. Understanding them before you write your first Pipeline saves the kind of debugging session that makes you question your career choices.

1. Fitting transformers on the full dataset before splitting. This is data leakage, and it is the most consequential mistake in the list. When StandardScaler.fit_transform() runs on all rows before train_test_split(), the scaler's mean and standard deviation incorporate test-set statistics. The model trains on features that were normalized using information it should never have accessed. Offline evaluation looks excellent because the test set was also normalized using its own statistics. Production data arrives with a different distribution and performance collapses. The fix is one line of code in the right position: call train_test_split() first, always.

2. Scaling features for tree-based models. This does not break anything — it just wastes engineering time and adds inference latency. Random Forest, XGBoost, LightGBM, and CatBoost make splits by comparing feature values against thresholds. The scale of those values is irrelevant. Applying StandardScaler to a gradient-boosted tree pipeline adds a preprocessing step to every inference call that contributes exactly nothing to predictive accuracy. The cost is low, but so is the benefit — and unnecessary complexity compounds over time.

3. Ignoring unknown categories in OneHotEncoder. Production data contains categories that did not exist when the model was trained. A new product category, a new geographic region, a new device type. Without handle_unknown='ignore', OneHotEncoder raises a ValueError and the inference service returns a 500 error for that request. With handle_unknown='ignore', unknown categories are encoded as all-zero vectors. The model has no information about the new category and defaults to its prior — not ideal, but the service stays up. Set this parameter by default and treat the appearance of unknown categories as a signal to evaluate whether retraining is needed.

4. Using imputation statistics computed on all available data. Sameroot cause as mistake one, different mechanism. SimpleImputer fit on the full dataset before splitting leaks test-set mean and median values into training. Inside a Pipeline, SimpleImputer.fit() is called only on training folds during cross-validation — this is exactly what the Pipeline is for. Outside a Pipeline, it is easy to call imputer.fit_transform(X) before the split without realizing the consequences.

# io.thecodeforge: The correct preprocessing pattern — no exceptions
#
# The wrong pattern (DO NOT DO THIS):
#   scaler = StandardScaler()
#   X_scaled = scaler.fit_transform(X)  # leaks test stats into training
#   X_train, X_test = train_test_split(X_scaled, ...)  # too late
#
# The right pattern:

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
import numpy as np

# Assume X has numerical columns [0,1] and categorical column [2]
# and some missing values scattered throughout
X = np.array([
    [25, 50000, 'Engineering'],
    [None, 80000, 'Marketing'],
    [45, None, 'Engineering'],
    [28, 62000, 'Marketing'],
    [35, 95000, None],
], dtype=object)

y = np.array([0, 1, 0, 1, 0])

# RULE 1: Split FIRST. Before any transformer is instantiated.
# This is the single most important line in any ML preprocessing script.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# RULE 2: Build preprocessing inside a Pipeline.
# Impute nulls before scaling — StandardScaler cannot handle NaN.
numerical_pipeline = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),  # fit on training only
    ('scaler', StandardScaler())                    # fit on training only
])

categorical_pipeline = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),  # handles null categories
    ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numerical_pipeline, [0, 1]),
        ('cat', categorical_pipeline, [2])
    ],
    remainder='drop'
)

# RULE 3: Wrap everything in a single Pipeline.
# cross_val_score() will call pipeline.fit() on each training fold,
# which internally calls preprocessor.fit() on that fold only.
# Test folds are transformed using training-fold parameters.
# Data leakage is structurally impossible inside this pattern.
forge_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression(max_iter=1000))
])

# Fit the final pipeline on all training data
forge_pipeline.fit(X_train, y_train)

# Transform test data using parameters learned from training only
test_accuracy = forge_pipeline.score(X_test, y_test)
print(f"Test accuracy (no leakage): {test_accuracy:.2f}")

# RULE 4: Save the entire Pipeline, not just the model.
# The scaler parameters and encoder vocabulary are part of the artifact.
import joblib
joblib.dump(forge_pipeline, 'forge_pipeline_v1.joblib')
print("Pipeline saved: preprocessing parameters and model travel together.")

Frequently Asked Questions