---
name: jmagly/dataset-version
source: https://app.decimal.ai/s/jmagly-dataset-version@1/SKILL.md
source_sha256: e8742c8b3210
---

# dataset-version

Create a versioned, reproducible training dataset: validate gates, compute splits and license, generate fixity, record provenance, and snapshot to immutable storage. This is the **final pipeline step** — a successful run yields a named, fixity-verified dataset version that downstream training, evaluation, or publication can consume deterministically.

Per ADR-022 D6, the YAML manifest is the source of truth; the sibling JSON is a regenerated export. Per ADR-022 D9, effective license is computed most-restrictive-wins across all contributing sources. Per ADR-022 D3, immutable storage is either a Fortemi archive snapshot or an `aiwg index` snapshot.

## When to Use

- **Final pipeline step**: after acquisition, synthesis, preference generation, filtering, and decontamination are complete — seal the dataset as a named version
- **Republishing**: when a prior version needs to be re-emitted with updated metadata (new decontamination report, new format exports) — always produces a new version, never mutates an existing one
- **Pre-release gate**: before shipping a dataset externally, to produce the fixity manifest and provenance record that receivers will verify

## Parameters

### `<version>` (required)
Dataset version identifier. MUST be valid CalVer (`YYYY.M.PATCH`, no leading zeros — e.g., `2026.4.0`) or SemVer (`MAJOR.MINOR.PATCH`). The manifest is written to `datasets/<version>.yaml`; collision with an existing file is a hard failure (versions are immutable).

### `--seed <n>` (optional)
Integer random seed that governs deterministic splits, shuffles, and any sampling. Default: `42`. Recorded in the manifest's `seed` field and in `reproduction_recipe` — the `dataset-reproduce` skill requires an exact match.

### `--split-ratios <train/val/test>` (optional)
Split ratios expressed as a comma-separated triple summing to `1.0`. Default: `0.8,0.1,0.1`. Computed split counts are written to `split_counts`. Test-set human-only enforcement (`synthetic_ratio.test == 0.0`) still applies unless explicitly overridden in the synthesis config.

### `--acknowledge-contamination` (optional)
Override the decontamination-gate lint rule (#843) when `decontamination-check` reports overlap above threshold. Requires an operator reason (prompted interactively if not provided via `--reason`). Appends an audit block to `manifest.ethical_considerations`, `.aiwg/activity.log`, and the decontamination report.

### `--acknowledge-license-risk` (optional)
Override the license-check lint rule (#837) when the computed effective license would block publication (incompatible SPDX combinations, copyleft conflicts, missing attribution chains). Same audit pattern as `--acknowledge-contamination`.

### `--storage <fortemi|aiwg-index>` (optional)
Immutable storage backend. Default: `fortemi` if a Fortemi archive is configured, else `aiwg-index`. Recorded in `storage_ref` per ADR-022 D3.

## Operation

1. **Validate gates** — run `memory-lint` with `license-check` and `decontamination-gate` rules against the candidate example set. On `ERROR`, block publication unless the matching `--acknowledge-*` flag is present with a recorded reason. On `WARN`, proceed but surface in the manifest's `ethical_considerations`.
2. **Compute splits** — resolve `split_counts` by applying `--split-ratios` to the candidate count with the declared `--seed` (same seed MUST produce same split given same inputs). Enforce `synthetic_ratio.test == 0.0` unless explicitly overridden.
3. **Compute effective license** — walk `sources[].license`, apply most-restrictive-wins resolution (per #837 and ADR-022 D9), and write to the top-level `license` field. If resolution is ambiguous (e.g., `GPL-2.0-only` vs `Apache-2.0`), escalate per `@agentic/code/addons/aiwg-utils/rules/human-authorization.md`.
4. **Compute synthetic ratio per split** — per-split human:synthetic mix written to `synthetic_ratio`. Test split is human-only by default (ADR-022 D10); validation typically human-only. Train ratio reflects actual synthesized fraction.
5. **Generate fixity manifest** — delegate to `integrity-verification` (media-curator) to emit `datasets/<version>-CHECKSUMS.sha256` covering every example record, config file, and report referenced by the manifest. Path is recorded in `fixity_manifest`.
6. **Record provenance** — delegate to `provenance-create` (sdlc-complete) to emit a W3C PROV-O record with the dataset version as `prov:Entity` and every source (`sources[].ref_id`), generator config, and upstream skill invocation in the derivation chain. Store at `provenance/dataset-<version>.jsonld`; record UUID in `provenance_record_id`.
7. **Snapshot to immutable storage** — either Fortemi (`mcp__memory-fortemi__manage_archives`) or `aiwg index snapshot` per ADR-022 D3. Record the resulting snapshot ID in `storage_ref` (exactly one of `fortemi_archive_id` or `aiwg_index_snapshot_id`).
8. **Write manifest** — render `datasets/<version>.yaml` per the schema at `@agentic/code/frameworks/training-complete/schemas/dataset-manifest.yaml`. Auto-export the sibling `datasets/<version>.json` immediately after. The YAML is source of truth; hand-edits to the JSON are forbidden.
9. **Log event** — append a `dataset-version` event to `.aiwg/activity.log` via `memory-log-append`, including version, snapshot ID, fixity manifest path, and any acknowledgement overrides.

## Overrides

Both `--acknowledge-contamination` and `--acknowledge-license-risk` follow the same audit pattern:

- **Manifest** — append a block to `ethical_considerations` naming the rule overridden, the operator, the timestamp, and the recorded reason
- **Activity log** — `memory-log-append` records a `dataset-version-override` event referencing the version and rule
- **Report annotation** — the upstream report (decontamination or license-check) is amended with a trailing "Acknowledged" section

Overrides do NOT suppress the underlying finding — they record an informed exception. Downstream consumers (model cards, datasheets) can surface these annotations.

## Output Files

| Path | Purpose |
|---|---|
| `datasets/<version>.yaml` | Dataset manifest (source of truth) |
| `datasets/<version>.json` | Auto-exported JSON mirror (regenerated, do not edit) |
| `datasets/<version>-CHECKSUMS.sha256` | SHA-256 fixity manifest covering every referenced artifact |
| `provenance/dataset-<version>.jsonld` | W3C PROV-O provenance record |

## Error Handling

Publication is atomic: if manifest write fails after snapshot is taken, the snapshot is rolled back (Fortemi archive delete or `aiwg index snapshot drop`) and no files remain in `datasets/`. Partial writes never leave a half-published version on disk. Any failure is logged as `dataset-version-failed` via `memory-log-append`.

On re-run after a failure, a fresh `<version>` is required — the same version number cannot be retried because the snapshot ID may differ.

## Delegation

- Fixity generation: `@agentic/code/frameworks/media-curator/skills/integrity-verification/SKILL.md`
- Provenance record: `@agentic/code/frameworks/sdlc-complete/skills/provenance-create/SKILL.md`
- Activity log: `@agentic/code/addons/semantic-memory/skills/memory-log-append/SKILL.md`
- Gate lint: `@agentic/code/addons/semantic-memory/skills/memory-lint/SKILL.md`
- Authorization for overrides: `@agentic/code/addons/aiwg-utils/rules/human-authorization.md`

## Examples

```bash
# Standard publication of a cleanly-gated dataset
dataset-version 2026.4.0

# Seed-deterministic publication with custom split
dataset-version 2026.4.0 --seed 1337 --split-ratios 0.85,0.075,0.075

# Publication with acknowledged contamination override
dataset-version 2026.4.0 --acknowledge-contamination \
  --reason "HumanEval overlap limited to 3 docstring comments in 10K corpus; test isolated"
```

## References

- REF-474 Dataset Versioning and Reproducibility
- REF-475 ML Reproducibility Checklist
- ADR-022 D6 — YAML primary + JSON auto-export
- ADR-022 D9 — Most-restrictive-wins license resolution
- ADR-022 D3 — Immutable storage (Fortemi archive OR aiwg index snapshot)
- Issue #844 — this skill
- Issue #837 — license-check lint rule
- Issue #843 — decontamination-gate lint rule
- `@agentic/code/frameworks/training-complete/schemas/dataset-manifest.yaml` — manifest schema