Dataset Splitting

Dataset Splitting

Overview

Train/validation/test splits are the single most important guardrail against overfitting and data leakage. A bad split invalidates everything downstream. Split once, lock the split, and never let test data influence any decision.

When to Use

Use this skill when:

• Preparing data for supervised learning.
• Designing evaluation protocols for model comparison.
• Setting up cross-validation folds.
• Ensuring no data leakage between splits.

Do not use for:

• Unsupervised learning evaluation — different rules apply.
• Time-series forecasting with backtesting — use darts or prophet skills.
• Data cleaning — use dataset-cleaning before splitting.

Split Strategies

1. Standard Random Split (IID)

from sklearn.model_selection import train_test_split

# Single split (fixed seed, stratified)
X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    stratify=y,
    random_state=42  # NEVER change this casually
)

# Train / validation / test (two-step)
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.1765, stratify=y_temp, random_state=42