Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/ase-experiments
Use when designing or auditing the evaluation of an ASE (IEEE/ACM Automated Software Engineering) paper, covering real subject systems, fair runnable tool baselines, task-matched effectiveness metrics, ablations that isolate a learned component, oracle and correctness validation, contamination-aware LLM handling, and provenance for mining.
Passed 17 of 22 test cases with the skill, and 6 of 22 without it.
and 2 more in the full report
18 fixed2 passed either way1 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 18 test cases the skill fixed. Same model, same prompt, with and without the skill.
To construct a ground-truth dataset of security commit classifications, a single graduate student manually categorized 500 commit logs without cross-validation or inter-rater agreement metrics. Critique our methodology using the critique layout.
| Checked | Without skill | With skill |
|---|---|---|
| Output includes the `[Threats]` tag. | Missed | Passed |
| Requires establishing a formal labeling protocol with inter-rater agreement metrics for manually coded data. | Passed | Passed |
The instructions your model reads when the skill loads.
Match the evidence to the automation's claim. ASE evaluations are judged on whether a tool or technique actually does what it claims on real subjects, compared fairly against the closest runnable automation. This is the axis reviewers weight most, and the one that most often becomes a Revision criterion.
Different automations demand different evidence:
| Automation claim | Evidence that matches | Common failure |
|---|---|---|
| Detection (bugs, smells, vulnerabilities) | Precision/recall/F on real defects with a defined ground truth | Synthetic-only defects; unclear ground truth |
| Generation / synthesis (tests, code, patches) | Validity of the produced artifact (compiles, passes, holds the property) | Similarity-to-reference proxy instead of validity |
| Repair | Verified behavior change: re-run + oracle; assertion/spec preservation | "Plausible patch" without an overfitting check |
| Localization / ranking | Rank-based effectiveness on real faults vs. alternatives | Cherry-picked programs; one metric only |
| Scalability / performance | Real-system sizes, wall-clock with a fair config | Toy inputs; unequal baseline budget |
If a learned or LLM component is involved, run an ablation that removes it and keeps the rest, so the marginal value of the design is visible. This is what defeats the "the model did it, not
test cases passed, vs 6 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.