Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →fcakyon/reproduce
End-to-end paper reproduction from arxiv URL through smoke runs to replication experiments. Handles missing or partial official code, missing training scripts, missing hyperparameters, and private datasets via similar-public-dataset substitution.
Passed 10 of 21 test cases with the skill, and 6 of 21 without it.
and 7 more in the full report
5 fixed5 passed either way11 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
This benchmark measured an earlier version of the skill.
A newer attempt on gemini-3.8-flash on 10/20/2026 did not produce a publishable comparison (not comparable). The snapshot above is the previously measured run.
One of 5 test cases the skill fixed. Same model, same prompt, with and without the skill.
Where should all the reproduction artifacts for arXiv paper 2305.11111 be stored, and what files should reside in the root of that folder?
| Checked | Without skill | With skill |
|---|---|---|
| Reproduction artifacts are organized inside a dedicated workspace folder named with the paper identifier. | Passed | Passed |
| Final evaluation metrics and comparison tables are saved in a dedicated results document in the workspace root. | Missed | Passed |
The instructions your model reads when the skill loads.
Reproducing an ML paper often means filling gaps the authors didn't ship, training scripts, hyperparameter tables, augmentation specifics, exact dataset splits. This skill walks seven stages from "I have an arxiv link" to "I have a replication run with measurable delta vs the paper's number."
Each stage has a separate reference file under references/ so this overview stays scannable.
The user just said any of:
| Stage | What | Reference |
|---|---|---|
| 1 | Paper acquisition (CLI + PDF → structured extract) | references/01-paper-fetch.md |
| 2 | Existing code discovery + inventory | references/02-code-clone.md |
| 3 | Gap analysis (extract every missing hyperparam from the prose) | references/03-gap-analysis.md |
| 4 | Implementation (uv venv, fill gaps, commit per gap) | references/04-implement.md |
| 5 | Dataset acquisition (HF datasets first; substitute if private) | references/05-dataset.md |
| 6 | Smoke runs (forward pass → 1 step → 20 iters) | references/06-smoke.md |
| 7 | Replication runs + comparison at paper's reported epochs | references/07-replicate.md |
Walk them in order. Each stage has its own success criteria; do not advance to the next until the current one passes.
For each paper reproduction, set up a dedicated workspace:
repro/<paper-arxiv-id>/
├── paper.md # structured extract from stage 1
├── inventory.md # what exists / missing from stage 2
├── gaps_filled.md # hyperparam table with provenance from stage 3
├── code/ # implementation from stage 4 (or cloned + extended)
├── data/ # dataset symlinks or actual data from stage 5
test cases passed, vs 6 / 21 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.