Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/asplos-reproducibility
Use when hardening an ASPLOS paper's results for independent repetition — pinning simulator versions and configs, recording kernel/firmware/BIOS state, packaging FPGA bitstreams and RTL, documenting hardware dependencies an evaluator may lack, and writing availability statements that match what the ACM badges will later require.
Passed 20 of 24 test cases with the skill, and 11 of 24 without it.
and 6 more in the full report
9 fixed11 passed either way4 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.
Evaluate our benchmarking setup and format the output as the standard six bracketed metrics block, ensuring the second line evaluates our execution logging script status.
| Checked | Without skill | With skill |
|---|---|---|
| The second line of the output starts with '[Capture automation]' | Missed | Passed |
The instructions your model reads when the skill loads.
Systems results decay fast: a kernel update, a microcode revision, or a silently
changed simulator default can move numbers by more than the paper's claimed margin.
Reproducibility work at ASPLOS is therefore state capture — recording the full
machine, model, and toolchain state behind every figure — done while the experiments
run, not reconstructed at camera-ready time. It also front-loads artifact
evaluation: the badge criteria (asplos-artifact-evaluation) are exactly a demand
that this state capture exists and works.
Maintain one ledger row per experimental platform, committed alongside results:
| Layer | Capture | Why it moves numbers |
|---|---|---|
| Silicon | CPU model + stepping, memory config/topology, device (e.g. CXL expander) firmware | Steppings differ in errata and prefetch behavior |
| Firmware/BIOS | Microcode revision; SMT, turbo, prefetcher, C-state, NUMA settings | Any one knob can swamp a 10% effect |
| OS | Kernel version + full config, relevant sysctls, mitigations state | Speculation mitigations alone shift syscall-heavy results |
| Toolchain | Compiler + flags, libraries, runtime versions | -O level and allocator choice are classic silent variables |
| Simulator | Exact commit, all config files, region/checkpoint method, warm-up length | Defaults change across releases without notice |
| FPGA | Board, toolchain version, constraints, bitstream hash, achieved clock | Re-synthesis at a different clock is a different experiment |
| Workloads | Suite versions, input sets, trace provenance and preprocessing | "SPEC" without input class is unrepeatable |
| Randomness | Seeds for any stochastic component + run counts | Needed for the dispersion numbers to mean anything |
Run at the start of every measurement session; store output next to the data:
#!/bin/sh
# state-capture.sh — commit this file and its output with each result set
uname -a; cat /proc/cmdline
grep -m1 'model name' /proc/cpuinfo; grep microcode /proc/cpuinfo | sort -u
cat /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | sort -u
cat /sys/devices/system/cpu/smt/control 2>/dev/null
numactl --hardware 2>/dev/null | head -5
cc --version | head -1
git -C "$SIM_DIR" rev-parse HEAD 2>/dev/null # simulator commit
sha256sum "$BITSTREAM" 2>/dev/null # FPGA bitstream identity
test cases passed, vs 11 / 24 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.