---
name: tangxiangru/reproducibility-check
source: https://app.decimal.ai/s/tangxiangru-reproducibility-check@1/SKILL.md
source_sha256: dd7d26d5b63f
---

# Reproducibility check

Stage 08 must write review and readiness artifacts under `workspace/reviews/`.
The purpose is to state honestly what a third party could reproduce from this
run's directory alone — not to assert that everything is fine.

The output of this check is a record of what was **verified**, what was
**not verified**, and what is **known not to reproduce**. All three categories
are legitimate. Only a fourth is not: recording an unchecked item as verified.

## Audit the run directory as a stranger would

Work from `runs/<run_id>/` and assume no access to the conversation, the
operator's memory, or the environment the run happened in.

| Question | Where the answer must live |
| --- | --- |
| What was the research goal? | `user_input.txt` |
| What was actually run? | `workspace/code/` — scripts, not descriptions |
| What data went in? | `workspace/data/`, with provenance for anything downloaded |
| What came out? | `workspace/results/`, indexed by `experiment_manifest.json` |
| How do the figures relate to the results? | a script under `workspace/code/` that regenerates them |
| What decisions were made and why? | the decision ledger in the stage summaries |
| Which stages actually completed? | `run_manifest.json` — see below |

## Read the manifest before claiming the run is complete

`run_manifest.json` distinguishes stages that were **approved** from stages
that were **skipped**. A skipped stage has `skipped: true` and a `skip_kind`
of `human` or `auto`; an `auto` skip means the stage exhausted its retry budget
in an unattended run with nobody in the loop, and its work was never done.

A readiness review that reports a run as complete when a stage was auto-skipped
is wrong in the most damaging direction. List every skipped stage in the
readiness artifact, with its `skip_reason`, and say what downstream claim is
weakened by the gap.

## The checklist

For each item, record `verified`, `not_verified`, or `fails`, with a one-line
reason. Never leave an item without evidence for its status.

- **Environment** — is there a record of the Python version and any
  dependencies? A script that imports a package the run never declares is not
  reproducible.
- **Determinism** — are seeds set and recorded? If results vary run to run,
  say by how much.
- **Data provenance** — for each file in `workspace/data/`, where did it come
  from, and can someone else obtain it? "Generated synthetically" is a complete
  answer only if the generator is in `workspace/code/`.
- **Results regeneration** — does a script take the data to the results, or is
  there a gap where a manual step happened?
- **Figures** — same question, from results to figures.
- **Claims to evidence** — does every claim in the manuscript point at a file
  in the run? `citation_verification.json` covers external claims; this covers
  the run's own.
- **Compute** — what hardware and how long. Venues ask; see the
  `venue-checklist` skill.

## Threats to validity

Write these in the paper's own voice, not as a disclaimer appendix nobody
reads. The ones that matter most in an AutoR run:

- **Single-seed results** presented as if replicated.
- **A skipped or thin stage** upstream of the claim — especially Stage 05.
- **Baselines that were not tuned** to the same effort as the proposed method.
- **Evaluation on the data the method was developed against**, with no held-out
  split.
- **Claims that outran the experiment**: check the Stage 02 hypothesis
  manifest against what Stage 06 actually measured, and flag anything the
  manuscript asserts more strongly than the analysis supports.

## Before you finish

- Every checklist item has a status and a reason.
- Every skipped stage is named, with its kind and reason.
- Nothing is marked verified that you did not actually run or read.
- The bundle under `workspace/artifacts/` contains what the checklist says it
  contains.