Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/colm-reproducibility
Use when hardening a COLM paper's reproducibility story — pinning open-weight checkpoints and tokenizers, handling API-model drift and deprecation honestly, versioning evaluation harnesses and prompts, disclosing compute, and writing availability statements that distinguish what is releasable from what is not.
Passed 19 of 22 test cases with the skill, and 19 of 22 without it.
2 fixed17 passed either way1 still failing2 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 2 test cases the skill fixed. Same model, same prompt, with and without the skill.
Our language model pre-training corpus contains web text scraped under restrictive terms that prohibit public distribution. Instead of withholding data information entirely, what specific proxy artifacts should be published to satisfy data disclosure expectations?
| Checked | Without skill | With skill |
|---|---|---|
| Recommends releasing pre-processing and data filtering code | Missed | Passed |
| Recommends publishing a dataset datasheet detailing data sources and summary statistics | Passed | Passed |
The instructions your model reads when the skill loads.
Reproducibility at a language-modeling venue is a different problem than at a classical ML venue, because the central artifact — the model — is often either too large to rerun or owned by someone else and changing weekly. COLM papers therefore need a layered story: what any reader can redo, what a funded lab can redo, and what will drift no matter what anyone does. Say which layer each claim lives in.
| Layer | Who can redo it | Your obligation |
|---|---|---|
| Exact | Anyone with the artifact | Pinned checkpoint + harness + prompts + decoding → same numbers (within stated sampling noise) |
| Budgeted | A lab with comparable compute | Full training recipe: data, hyperparameters, schedule, hardware, wall-clock, known failure modes |
| Drifting | Nobody, eventually | API-model results: version string, query dates, and an expiry-honest caveat in the paper |
A paper whose headline claim lives entirely in the drifting layer needs to say so on the page where the claim is made — and ideally to anchor the finding on at least one open-weight model so the exact layer is not empty.
Closed models are legitimate scientific objects at COLM — much of the field studies them — but they demand a specific discipline:
# An open-weight result is pinned when all four hashes exist in the paper/repo:
MODEL_REV=8f3c21a # checkpoint revision, not just the model name
test cases passed, vs 19 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.