---
name: aperivue-version-dataset
description: "Use when you must prove an analysis ran on the intended data or lock a dataset version. Builds a deterministic content-hash manifest (file SHA-256, schema, per-column value hashes), verifies later copies against it for drift, and diffs two manifests."
license: "MIT"
metadata:
  decimalai-skill-id: "07772b8c-2c3f-4130-86fc-937bcebc8351"
  decimalai-version-id: "303cdb93-e199-419a-b855-7a14e44c798b"
  decimalai-version: "2"
  decimalai-body-sha256: "89e117d249e33cd11d34967623f3b7e670d64ee3626d4739a4e114e983c2e0d7"
  decimalai-name: "aperivue/version-dataset"
  decimalai-format: "1"
  decimalai-invocation: "model"
  decimalai-category: "data"
  decimalai-source: "https://app.decimal.ai/s/aperivue-version-dataset@2/SKILL.md"
---
# Version Dataset Skill

An analysis must run on the data it claims to, with a fixed seed. This skill makes drift between
runs loud instead of silent: it records a deterministic fingerprint (file SHA-256 plus, for
tabular files, schema and per-column value hashes; no timestamp unless explicitly passed) so a
later run can *prove* the inputs are unchanged. It never alters data, and manifests hold hashes,
not the data itself. Provenance notes are in English.

## Deterministic Script

```bash
# Build a manifest (record the analysis seed + provenance)
python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" manifest data.csv \
  --out manifest.json --seed 42 --provenance "KNHANES 2018 extract v1"

# Verify a later copy against it (CI / pre-analysis gate)
python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" verify --manifest manifest.json --strict

# Compare two manifests (what changed between versions)
python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" diff --old v1.json --new v2.json
```

File hashing is stdlib-only; tabular schema/column hashing uses pandas when present.
`--ignore-cols` excludes volatile columns; `--base` makes manifest keys relative.

## Workflow

### Step 1: Lock the version (gate)

Build the manifest at the moment the dataset is frozen for analysis. **Gate:**
confirm with the user the seed and provenance note are correct before locking —
the manifest is the record they will cite as "this is the data the results came from."
The provenance note is the user's text; never invent it.

### Step 2: Verify before each run (gate)

Before re-running an analysis (or in CI), `verify --strict`. Never say a dataset is unchanged
without running `verify`. **Gate:** if drift is reported (`MANIFEST_DRIFT`, exit 1), stop and
show the user the drift report exactly as computed — never downplay a changed column hash. Do not
proceed on changed data without their explicit acknowledgement and a re-lock. Silent re-run on
drifted data is the failure this skill exists to prevent.

Read `${CLAUDE_SKILL_DIR}/references/manifest_schema.md` before interpreting drift — it defines
each drift category. A CSV/TSV file is compared on its logical content (schema + per-column
value hashes), not raw bytes, so re-quoting, reordering columns, or an `--ignore-cols` volatile
column does not trip a false drift. Binary tabular files (Parquet/Stata/SAS/Excel) are also
checked at the byte level, since labels, metadata and other sheets are not in the column hashes,
unless an `--ignore-cols` column is present in the file (its changes alter the bytes).

### Step 3: Diff across versions

When a dataset is intentionally updated, `diff` the old and new manifests and
present the change set (added/removed/changed columns, row-count delta) so the
user can record what changed and re-lock. **Gate:** the user approves the new
version before it replaces the locked one.

## Non-Deterministic Artifacts

Do not put PPTX/DOCX or figure binaries under strict byte verification — they embed timestamps
or render metadata and change on every build. Manifest only the deterministic inputs and tabular
outputs (data files, result CSVs), or use `--ignore-cols` for volatile columns; the policy is in
`references/manifest_schema.md`. Each bundled `demo/*/` carries a `manifest.lock.json` (input data
+ deterministic result tables) that `verify --strict` checks; the reference gives the demo
command.

## Scope

- Run `/deidentify` before a manifest is shared: example values are not stored, but provenance
  notes may carry context.
- Cleaning, profiling or de-identifying the data → `/clean-data`, `/generate-codebook`,
  `/deidentify`. `/generate-codebook` documents *what* is in the data; this skill locks *which
  version*.