ASE Reproducibility

ASE Reproducibility

Build the reproducibility story at data-collection time, not at submission. ASE requires a mandatory Data Availability Statement in the paper and expects an anonymized, runnable artifact at review time; automated-SE artifacts are usually tools, so "runnable" means a reviewer can actually execute the automation on stated subjects. What is not pinned when you collect it cannot be reconstructed later.

The mandatory Data Availability Statement

• Required, placed after the Conclusions and inside the 10-page limit (it is not free appendix space).
• State what exists — the tool, the dataset, the subject systems, the scripts, the logs — and where it will live after acceptance (an archival DOI target).
• Provide an anonymized link or upload now; "available upon request" reads as a scored weakness, not a neutral placeholder.
• Match the statement to what the archive actually contains — an overclaiming statement is worse than a modest, honest one.

Anonymized-but-runnable tools

• Re-host the tool and dataset behind an anonymizing service; strip repository owner, commit author metadata, and any path revealing your identity (/home/<you>/, institutional URLs).
• Include a minimal run path: exact commands, expected inputs, and a small sample so a reviewer can execute the automation without your machine.
• Pin the environment: dependencies with versions, a container or lockfile, and the exact tool commit — automated-SE tools rot fast against moving toolchains.

Provenance pinning (do this at collection time)

For the tool:

• Exact commit SHA, build instructions, dependency versions, and configuration/flags used in the experiments (including seeds for randomized components).

For subject systems and datasets:

• Names, versions, and SHAs of every subject; the corpus extraction date; query/filter criteria; and any manual labeling protocol with inter-rater agreement.
• A regeneration script and a versioned snapshot — live scraping re-samples a moving target.

For LLM-based components: