Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/atc-artifact-evaluation
Use when packaging an ATC (ACM SIGOPS Annual Technical Conference, formerly USENIX ATC) artifact for the USENIX-lineage evaluation scheme — earning the Artifacts Available, Artifacts Functional, and Results Reproduced badges from the Artifact Evaluation Committee on its separate post-acceptance deadline, with evaluator-proof documentation and a turnkey path to the paper's numbers.
Passed 19 of 22 test cases with the skill, and 17 of 22 without it.
and 2 more in the full report
4 fixed16 passed either way1 still failing2 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
Our USENIX ATC paper was accepted yesterday, and I'm putting together our artifact package for the evaluation committee. We built a custom distributed key-value store with RDMA support and included Dockerfiles, run scripts, and evaluation data. Could you review our readiness? Please output a summary specifying the target badges we should aim for, the artifact role (review or badge release), the required package contents breakdown, an assessment of whether an evaluator can complete an install and demo in under ten minutes on a fresh host, verification of claim-to-script mapping with small-scale fallback support, and a prioritized list of action items to fix before uploading.
| Checked | Without skill | With skill |
|---|---|---|
| The response identifies the artifact role as post-acceptance badge evaluation rather than anonymous review. | Missed | Passed |
| The response recommends a small-scale fallback mode for evaluators without RDMA hardware. | Passed | Passed |
The instructions your model reads when the skill loads.
Use this for the artifact track. ATC follows the shared USENIX/SIGOPS artifact-evaluation scheme: an Artifact Evaluation Committee (AEC) runs a separate, post-acceptance process through HotCRP and awards up to three badges — Artifacts Available, Artifacts Functional, and Results Reproduced. Two things to internalize: badges are earned by evaluators actually using your package, and the review artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact (de-anonymized, permanently archived). ATC's artifact culture is strong and high-participation — for USENIX ATC '25, most evaluated artifacts earned Functional and a large majority earned Reproduced.
| Badge | What it certifies | What earns it |
|---|---|---|
| Artifacts Available | The artifact is permanently, publicly retrievable | Deposit in a DOI-issuing archive (Zenodo, figshare, Software Heritage) |
| Artifacts Functional | It is documented, complete, and exercisable — it runs and does what the paper says | Clean-machine install, a demo, scripts + data to run the experiments |
| Results Reproduced | An evaluator reproduced the paper's main results | A turnkey path from the artifact to the headline figures/tables |
Available is low-cost, high-value (archive the package). Functional and Reproduced require the evaluator's own run to succeed, so the failure mode is always "did not run on their machine," never "the idea was weak."
| Claim type | First thing inspected | Common failure caught |
|---|---|---|
| A built system / mechanism | The README and one install/run command | Undocumented deps; only-works-on-authors'-hardware |
| A performance result | The scripts that turn a run into the paper's figures | Numbers in the PDF no script reproduces |
| A result needing specific hardware (RDMA/NVMe/NIC) | The documented host requirements + a small-scale mode | Requires unavailable hardware; no fallback |
| A trace-driven study | The traces + the processing scripts | Query shipped, data missing; provenance unpinned |
Assume the evaluator has a bounded time budget on a clean machine and may lack your exact hardware. Design the first ten minutes to succeed and provide a reduced-scale path.
[Container] a Dockerfile or pinned environment; document any host/hardware the container cannot abstract
[README] one-screen orientation: what it is, install, run the demo, reproduce each claim, expected runtime/outputs
test cases passed, vs 17 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.