ML Classification Pipeline

A disciplined, reproducible tabular classification workflow that avoids the usual foot-guns (leakage, tuning on the test set, accuracy-only on imbalanced data).

When to use

Labeled tabular data + a categorical target the user wants to predict: binary or multiclass. Not for regression (continuous target), unstructured data (text/images), or pure data exploration (use eda-report first).

Workflow

1. Frame it. Confirm the target column, whether it's binary or multiclass, and

the class balance. Decide the primary metric up front based on the cost of errors (ROC-AUC / F1 / recall for imbalanced or asymmetric-cost problems; accuracy only when classes are balanced and errors are symmetric).

1. Split first. train_test_split with stratify=y and a fixed random_state

before any fitting. The test set is touched exactly once, at the end.

1. Preprocess inside a Pipeline. Use ColumnTransformer:

numeric → impute + scale; categorical → impute + one-hot (or ordinal where ordered). Everything fit on train folds only — this is what prevents leakage.

1. Baseline. Fit a simple, interpretable baseline (LogisticRegression, or

DummyClassifier for the floor). Record cross-validated scores. Every later model must beat this to justify its complexity.

1. Candidate models. Typically LogisticRegression, RandomForest, and a gradient

booster (HistGradientBoosting / XGBoost / LightGBM if available). Compare via stratified k-fold CV on the training set using the primary metric.

1. Tune. GridSearchCV / RandomizedSearchCV (or Optuna for big spaces) with

stratified CV, optimizing the primary metric. Handle imbalance via class_weight='balanced' or resampling inside CV folds — never resample the whole dataset before splitting.

1. Evaluate once. Refit the best pipeline on full train, predict on the held-out

test set, and report the full picture: accuracy, precision/recall/F1 (per class + macro), ROC-AUC (or PR-AUC for heavy imbalance), and the confusion matrix. State the operating threshold if it was tuned.

1. Explain. Feature importance (permutation importance preferred over impurity;

SHAP if the user wants per-prediction explanations). Note caveats and any drift between CV and test performance.

Output

A runnable, reproducible script/notebook with fixed seeds, plus a short results summary: chosen metric and why, baseline-vs-tuned table, held-out test metrics, confusion matrix, top features, and honest limitations. Persist the fitted pipeline (joblib.dump) if the user wants to reuse it.

Notes / edge cases

• Report CV and held-out numbers. A big gap signals overfitting or leakage.
• Quote metrics with their context (e.g. "ROC-AUC 0.85 on a 30% held-out test set,

stratified, seed=42"), not as bare numbers.

• Small datasets: prefer repeated stratified CV and lean models; don't over-tune.
• Severe imbalance: lead with PR-AUC / recall, not accuracy.