Dataset Versioning

Dataset Versioning

Overview

Datasets are artifacts that evolve. Versioning turns "which version did we use?" from a forensic investigation into a one-line answer. Every released dataset gets a checksum, a manifest, and a semantic version tag.

When to Use

Use this skill when:

• Creating, cleaning, or updating a dataset that others (or future-you) depend on.
• Tracking provenance: what source data, code, and parameters produced this dataset.
• Collaborating on datasets where multiple versions coexist.
• Reproducing past experiments that depend on a specific dataset version.

Do not use for:

• One-off exploration that won't be reused.
• Interim files that are fully regenerable from versioned sources.
• Model versioning — use mlflow or wandb skills.

Versioning Workflow

1. Initialize Version Tracking

# DVC (Data Version Control) — recommended for datasets > 10 MB
pip install dvc
dvc init
git commit -m "Initialize DVC"

Or minimal filesystem-based versioning:

import hashlib
import json
from pathlib import Path
from datetime import datetime, timezone

DATASET_ROOT = Path("datasets")
VERSION_REGISTRY = DATASET_ROOT / "versions.jsonl"