Reproducibility check

Stage 08 must write review and readiness artifacts under workspace/reviews/. The purpose is to state honestly what a third party could reproduce from this run's directory alone — not to assert that everything is fine.

The output of this check is a record of what was verified, what was not verified, and what is known not to reproduce. All three categories are legitimate. Only a fourth is not: recording an unchecked item as verified.

Audit the run directory as a stranger would

Work from runs/<run_id>/ and assume no access to the conversation, the operator's memory, or the environment the run happened in.

| Question | Where the answer must live | | --- | --- | | What was the research goal? | user_input.txt | | What was actually run? | workspace/code/ — scripts, not descriptions | | What data went in? | workspace/data/, with provenance for anything downloaded | | What came out? | workspace/results/, indexed by experiment_manifest.json | | How do the figures relate to the results? | a script under workspace/code/ that regenerates them | | What decisions were made and why? | the decision ledger in the stage summaries | | Which stages actually completed? | run_manifest.json — see below |

Read the manifest before claiming the run is complete

run_manifest.json distinguishes stages that were approved from stages that were skipped. A skipped stage has skipped: true and a skip_kind of human or auto; an auto skip means the stage exhausted its retry budget in an unattended run with nobody in the loop, and its work was never done.

A readiness review that reports a run as complete when a stage was auto-skipped is wrong in the most damaging direction. List every skipped stage in the readiness artifact, with its skip_reason, and say what downstream claim is weakened by the gap.

The checklist

For each item, record verified, not_verified, or fails, with a one-line reason. Never leave an item without evidence for its status.

• Environment — is there a record of the Python version and any

dependencies? A script that imports a package the run never declares is not reproducible.

• Determinism — are seeds set and recorded? If results vary run to run,

say by how much.

• Data provenance — for each file in workspace/data/, where did it come

from, and can someone else obtain it? "Generated synthetically" is a complete answer only if the generator is in workspace/code/.

• Results regeneration — does a script take the data to the results, or is

there a gap where a manual step happened?

• Figures — same question, from results to figures.
• Claims to evidence — does every claim in the manuscript point at a file

in the run? citation_verification.json covers external claims; this covers the run's own.

• Compute — what hardware and how long. Venues ask; see the

venue-checklist skill.

Threats to validity

Write these in the paper's own voice, not as a disclaimer appendix nobody reads. The ones that matter most in an AutoR run:

• Single-seed results presented as if replicated.
• A skipped or thin stage upstream of the claim — especially Stage 05.
• Baselines that were not tuned to the same effort as the proposed method.
• Evaluation on the data the method was developed against, with no held-out

split.

• Claims that outran the experiment: check the Stage 02 hypothesis

manifest against what Stage 06 actually measured, and flag anything the manuscript asserts more strongly than the analysis supports.

Before you finish

• Every checklist item has a status and a reason.
• Every skipped stage is named, with its kind and reason.
• Nothing is marked verified that you did not actually run or read.
• The bundle under workspace/artifacts/ contains what the checklist says it

contains.