---
name: thomson-li/ml-classification-pipeline
source: https://app.decimal.ai/s/thomson-li-ml-classification-pipeline@1/SKILL.md
source_sha256: 373c4e0f049c
---

# ML Classification Pipeline

A disciplined, reproducible tabular classification workflow that avoids the usual
foot-guns (leakage, tuning on the test set, accuracy-only on imbalanced data).

## When to use

Labeled tabular data + a categorical target the user wants to predict: binary or
multiclass. **Not** for regression (continuous target), unstructured data (text/images),
or pure data exploration (use `eda-report` first).

## Workflow

1. **Frame it.** Confirm the target column, whether it's binary or multiclass, and
   the **class balance**. Decide the primary metric *up front* based on the cost of
   errors (ROC-AUC / F1 / recall for imbalanced or asymmetric-cost problems; accuracy
   only when classes are balanced and errors are symmetric).

2. **Split first.** `train_test_split` with `stratify=y` and a fixed `random_state`
   **before** any fitting. The test set is touched exactly once, at the end.

3. **Preprocess inside a Pipeline.** Use `ColumnTransformer`:
   numeric → impute + scale; categorical → impute + one-hot (or ordinal where
   ordered). Everything fit on **train folds only** — this is what prevents leakage.

4. **Baseline.** Fit a simple, interpretable baseline (LogisticRegression, or
   DummyClassifier for the floor). Record cross-validated scores. Every later model
   must beat this to justify its complexity.

5. **Candidate models.** Typically LogisticRegression, RandomForest, and a gradient
   booster (HistGradientBoosting / XGBoost / LightGBM if available). Compare via
   stratified k-fold CV on the training set using the primary metric.

6. **Tune.** `GridSearchCV` / `RandomizedSearchCV` (or Optuna for big spaces) with
   stratified CV, optimizing the primary metric. Handle imbalance via
   `class_weight='balanced'` or resampling **inside** CV folds — never resample the
   whole dataset before splitting.

7. **Evaluate once.** Refit the best pipeline on full train, predict on the held-out
   test set, and report the full picture: accuracy, precision/recall/F1 (per class +
   macro), ROC-AUC (or PR-AUC for heavy imbalance), and the confusion matrix. State
   the operating threshold if it was tuned.

8. **Explain.** Feature importance (permutation importance preferred over impurity;
   SHAP if the user wants per-prediction explanations). Note caveats and any drift
   between CV and test performance.

## Output

A runnable, reproducible script/notebook with fixed seeds, plus a short results
summary: chosen metric and *why*, baseline-vs-tuned table, held-out test metrics,
confusion matrix, top features, and honest limitations. Persist the fitted pipeline
(`joblib.dump`) if the user wants to reuse it.

## Notes / edge cases

- **Report CV *and* held-out numbers.** A big gap signals overfitting or leakage.
- Quote metrics with their context (e.g. "ROC-AUC 0.85 on a 30% held-out test set,
  stratified, seed=42"), not as bare numbers.
- Small datasets: prefer repeated stratified CV and lean models; don't over-tune.
- Severe imbalance: lead with PR-AUC / recall, not accuracy.