--- name: brycewang-stanford/corl-reproducibility source: https://app.decimal.ai/s/brycewang-stanford-corl-reproducibility@1/SKILL.md source_sha256: 4a2a78cdef3c --- # CoRL Reproducibility Robot-learning papers have a split reproducibility problem: the *training* half is software and can in principle be rerun anywhere, while the *hardware* half is a physical setup nobody else owns. Strong CoRL papers treat these halves differently — the software half is made **rerunnable**, the hardware half is made **auditable** — and say plainly which is which. ## The rerunnable half: simulation and training The silent reproducibility killers in this field are version-shaped: - **Simulator versions change physics.** Contact solvers, default damping, and collision margins shift between releases of MuJoCo, Isaac, PyBullet, and friends; a policy's success rate is a function of the simulator build. Pin the exact version and any physics-relevant flags. - **Environment wrappers drift.** Task-suite repositories (benchmark forks, custom reward shims) move under you; record the commit hash of every env repo, not just your own. - **GPU nondeterminism.** cuDNN autotuning and atomics make bit-identical training runs unrealistic — so define reproducibility at the *distribution* level: same configs + fresh seeds should land inside your reported seed spread. ```yaml # repro-manifest.yaml — ship at the repo root; one block per results table table_3: code_commit: "" # your repo @ exact hash env_suite: {repo: "", commit: "", sim: "mujoco==X.Y.Z", physics_flags: []} training: {config: "configs/table3.yaml", seeds: [0,1,2,3,4], gpu: "1xA100-80GB", hours: 14} data: {demos: "dataset-v2 (1,204 episodes)", url_or_status: "", license: ""} checkpoints: {released: true, path: "ckpts/table3/", selection_rule: "last epoch, no eval peeking"} evaluation: {script: "eval/run.py", episodes_per_task: 50, init_state_list: "eval/states.json"} expected_tolerance: "per-task success within ±4 pts of Table 3 mean" ``` The `selection_rule` line matters more than it looks: checkpoint selection via test-set peeking is the field's quiet irreproducibility engine, and stating the rule is the cheapest credibility purchase available. ## The auditable half: hardware Nobody will re-run your robot, so the goal is that a skeptical expert could *verify the experiment happened as described* and *rebuild an equivalent rig*: | What to document | Why an auditor needs it | |---|---| | Robot model, end-effector, firmware/driver versions | Behavior differs across firmware, not just robots | | Sensor models, mounting poses, calibration procedure | Camera extrinsics silently dominate visuomotor results | | Control interface: frequency, action space, safety filters | "30 Hz end-effector deltas" vs "torque control" are different papers | | Deployed compute + inference latency | Policy behavior is latency-dependent | | Scene inventory: objects (make/size), fixtures, lighting | Enables an equivalent-rig rebuild and honest comparison | | Raw episode logs and unedited evaluation video | The audit trail for every printed success rate | Log every evaluation episode at capture time (timestamped video plus a CSV of outcomes); retrofitting an audit trail after reviews ask for it is impossible. ## Data and checkpoint release - Demonstration datasets are results: release episode counts, collection method (teleop rig, scripted, crowdsourced), operator count, and filtering rules. - Released checkpoints let others reproduce *evaluation* even when training is too expensive to repeat — for large policies this is often the highest-value artifact you can ship. - If data or weights cannot be released (proprietary platform, human-subject footage), say so in the paper with the reason, and release what remains: configs, eval scripts, sim environments, and metrics logs. A precise partial-release statement outperforms a vague "code available upon request." ## During review vs after acceptance - During review everything must be anonymous (see `corl-submission`): anonymized repo mirrors, no lab-identifying video, no cloud buckets with named projects. - The 2026 camera-ready pipeline has a sharp constraint: **PMLR does not accept video as supplementary material**, so post-acceptance videos, code, and data live on external hosting (project site, GitHub, archive), linked from the main text (corl.org author instructions, read 2026-07-08). Plan the public homes of artifacts before the October camera-ready deadline, and prefer DOI-stamped archives for anything you cite as permanent. - No formal reproducibility checklist was verified for the 2026 cycle in this pack (待核实) — but reviewer expectations enforce one informally; use the manifest above regardless of what the form requires. ## Availability statement patterns ```text Strong: "Code, training configs, evaluation scripts, and the 1,204-episode teleop dataset: . Checkpoints for Tables 2-4: . Hardware evaluations are documented in Appendix C (rig spec, logs, uncut video); the platform itself cannot be redistributed." Weak: "Code will be released upon acceptance." (unverifiable promise) Broken: "Results reproducible with standard settings." (no artifact at all) ``` ## Pre-submission audit ```text [ ] repro-manifest present; one block per headline table [ ] Simulator/env/driver versions + commits pinned everywhere [ ] Seed policy stated; checkpoint selection rule stated [ ] Hardware rig spec complete enough for an equivalent rebuild [ ] Episode-level logs + uncut eval video archived internally [ ] Data/checkpoint release plan with named blockers, if any [ ] Anonymous during review; external hosting plan ready for camera-ready ``` Verify the live cycle's supplementary and camera-ready rules at https://www.corl.org/contributions/instruction-for-authors before promising any artifact channel — PMLR-side constraints and CoRL-side forms both change yearly.