ASE Experiments

Match the evidence to the automation's claim. ASE evaluations are judged on whether a tool or technique actually does what it claims on real subjects, compared fairly against the closest runnable automation. This is the axis reviewers weight most, and the one that most often becomes a Revision criterion.

Start from the claim shape

Different automations demand different evidence:

| Automation claim | Evidence that matches | Common failure | |---|---|---| | Detection (bugs, smells, vulnerabilities) | Precision/recall/F on real defects with a defined ground truth | Synthetic-only defects; unclear ground truth | | Generation / synthesis (tests, code, patches) | Validity of the produced artifact (compiles, passes, holds the property) | Similarity-to-reference proxy instead of validity | | Repair | Verified behavior change: re-run + oracle; assertion/spec preservation | "Plausible patch" without an overfitting check | | Localization / ranking | Rank-based effectiveness on real faults vs. alternatives | Cherry-picked programs; one metric only | | Scalability / performance | Real-system sizes, wall-clock with a fair config | Toy inputs; unequal baseline budget |

Real subject systems

• Use real software — open-source projects, real bug/defect datasets, real CI logs — not toy

programs you constructed to make the tool look good.

• Report subject provenance: names, versions/commit SHAs, sizes, and the extraction date.

Reviewers reproduce from this.

• Justify subject selection and disclose exclusions; self-selected subjects are the classic

external-validity threat.

Fair, runnable tool baselines

• Compare against the closest runnable automation, configured at an equal, documented budget

(time, iterations, tuning, seeds). ASE reviewers routinely rerun or scrutinize baselines.

• Pin baseline versions/commits and note reimplementation vs. original.
• If no tool baseline exists, construct a defensible non-trivial baseline (a static rewrite, a

random or heuristic variant) rather than comparing only to "nothing."

Ablations that isolate the automation

If a learned or LLM component is involved, run an ablation that removes it and keeps the rest, so the marginal value of the design is visible. This is what defeats the "the model did it, not your technique" objection and keeps the paper ASE-shaped rather than ML-shaped.

Oracles and correctness

• State the oracle explicitly: how do you know a generated test is meaningful, or a repair is

correct? Re-execution, differential testing, formal checks, or human audit — name it.

• For repair/synthesis, guard against overfitting to the evaluation oracle (e.g., patches that

pass the given tests but break behavior): report a held-out or manual correctness check.

Statistics and effect sizes

• Report effect sizes and dispersion (confidence intervals, non-parametric tests where

appropriate), not just point estimates or a single accuracy number.

• For randomized techniques (search-based, sampling, LLM temperature > 0), report repeated runs

with variance and fix/seed the randomness for the artifact.

Contamination-aware LLM handling

• Record model identifiers and dates; a model updated between runs invalidates comparisons.
• Consider training-data contamination: benchmarks the model may have seen inflate results —

report on held-out or post-cutoff subjects where feasible, and say so.

• Cache raw model outputs so the artifact reproduces rather than re-samples a live API.

Mining and dataset provenance

• Pin repository SHAs, the corpus extraction date, query/filter criteria, and any labeling

protocol with inter-rater agreement for manually coded data.

• Version the dataset and describe how to regenerate it; a package that needs live scraping

re-samples a moving target.

Evaluation audit checklist

text
[Claim-evidence] each claim -> a matching metric on real subjects (not a proxy)
[Subjects] real, provenance-pinned, selection justified, exclusions disclosed
[Baselines] closest runnable tool, version pinned, equal documented budget
[Ablation] learned/LLM component isolated; marginal value of the design shown
[Oracle] correctness defined; overfitting-to-oracle checked
[Stats] effect sizes + dispersion; repeated runs for randomized methods
[LLM] model IDs/dates recorded; contamination considered; outputs cached
[Repro] provenance pinned; dataset/tool versioned for the artifact

Output format

text
[Automation claim] detection / generation / repair / localization / scalability
[Evidence match] metric(s) that fit the claim, on real subjects
[Baseline fairness] closest tool, budget parity, versions
[Ablation + oracle] learned-component ablation present; correctness oracle stated
[Threats] subject selection / oracle validity / baseline fairness / contamination — bounded how?