Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/asplos-experiments
Use when designing or auditing the evaluation of an ASPLOS paper — choosing among real silicon, FPGA prototypes, and simulators with cycle-accuracy caveats stated, selecting workload suites and baselines that hold up across three communities, attributing wins via ablation, and reporting energy, area, and overhead honestly.
Passed 17 of 22 test cases with the skill, and 12 of 22 without it.
and 5 more in the full report
5 fixed12 passed either way5 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 5 test cases the skill fixed. Same model, same prompt, with and without the skill.
I'm preparing our ASPLOS paper submission draft evaluating a novel CXL memory-tiering architecture across hardware hints and OS paging policies. Could you audit our evaluation section draft? Please organize your feedback using the standard six-part assessment report format covering: 1) Matrix verification linking claims to instruments and text sections, 2) Instrument capability audits to spot overstated claims, 3) Simulator hygiene and validation against silicon, 4) Baseline strength and tuning verification, 5) Cross-layer ablation completeness, and 6) Adverse results and saturation boundary disclosures.
| Checked | Without skill | With skill |
|---|---|---|
| The response output begins with or contains the exact bracketed header line '[Matrix]' | Missed | Passed |
| The response includes all six bracketed diagnostic lines: '[Matrix]', '[Instrument audit]', '[Simulator hygiene]', '[Baseline strength]', '[Attribution]', and '[Adverse results]' | Missed | Passed |
The instructions your model reads when the skill loads.
An ASPLOS evaluation answers to three communities at once: architects who will audit the modeling, OS people who will audit the workload realism, and PL people who will audit what the software layer actually does. The section's core discipline is matching each claim to an instrument whose error model can carry it — and saying what that error model is.
| Instrument | What it can prove | What it cannot | Must be reported |
|---|---|---|---|
| Real silicon | End-to-end effects, OS interactions, true tails | Designs needing hardware that doesn't exist | CPU/stepping, kernel + config, microcode, BIOS knobs (SMT/turbo/prefetchers), memory topology |
| FPGA prototype | Feasibility, cycle behavior of new logic at the prototype's clock | Absolute performance of an ASIC-class part | Board, clock, resource utilization, what was scaled down and why |
| Cycle-level simulator (e.g. gem5-class) | Relative effects of microarchitectural change under stated configs | Anything outside modeled fidelity — I/O, OS noise, firmware behavior are commonly stylized | Simulator + exact version/commit, config files, warm-up and region-selection method, validation against a real machine where possible |
| Analytical/energy models (McPAT-class, first-order area) | Trend-level energy/area comparisons | Absolute mW or mm² as truth | Model version, technology node assumptions, and the claim written as trend not absolute |
The cardinal sin is a claim-instrument mismatch: absolute latency claims from an unvalidated simulator, or OS-interaction claims from a user-space harness. Rapid and full reviewers both hunt for it.
When simulation carries a claim, the paper must state: which structures are modeled in detail vs stylized; how simulation regions were chosen (full runs, checkpoints, sampled regions à la SimPoint-style methodology); how long the warm-up was; and — strongest of all — a validation experiment showing the simulator tracks a real machine on a measurable subset. A one-paragraph validation against silicon buys credibility that no amount of extra benchmarks can.
test cases passed, vs 12 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.