ASPLOS Reproducibility

Systems results decay fast: a kernel update, a microcode revision, or a silently changed simulator default can move numbers by more than the paper's claimed margin. Reproducibility work at ASPLOS is therefore state capture — recording the full machine, model, and toolchain state behind every figure — done while the experiments run, not reconstructed at camera-ready time. It also front-loads artifact evaluation: the badge criteria (asplos-artifact-evaluation) are exactly a demand that this state capture exists and works.

The state ledger

Maintain one ledger row per experimental platform, committed alongside results:

| Layer | Capture | Why it moves numbers | |---|---|---| | Silicon | CPU model + stepping, memory config/topology, device (e.g. CXL expander) firmware | Steppings differ in errata and prefetch behavior | | Firmware/BIOS | Microcode revision; SMT, turbo, prefetcher, C-state, NUMA settings | Any one knob can swamp a 10% effect | | OS | Kernel version + full config, relevant sysctls, mitigations state | Speculation mitigations alone shift syscall-heavy results | | Toolchain | Compiler + flags, libraries, runtime versions | -O level and allocator choice are classic silent variables | | Simulator | Exact commit, all config files, region/checkpoint method, warm-up length | Defaults change across releases without notice | | FPGA | Board, toolchain version, constraints, bitstream hash, achieved clock | Re-synthesis at a different clock is a different experiment | | Workloads | Suite versions, input sets, trace provenance and preprocessing | "SPEC" without input class is unrepeatable | | Randomness | Seeds for any stochastic component + run counts | Needed for the dispersion numbers to mean anything |

Scripted capture beats remembered capture

Run at the start of every measurement session; store output next to the data:

bash
#!/bin/sh
# state-capture.sh — commit this file and its output with each result set
uname -a; cat /proc/cmdline
grep -m1 'model name' /proc/cpuinfo; grep microcode /proc/cpuinfo | sort -u
cat /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | sort -u
cat /sys/devices/system/cpu/smt/control 2>/dev/null
numactl --hardware 2>/dev/null | head -5
cc --version | head -1
git -C "$SIM_DIR" rev-parse HEAD 2>/dev/null   # simulator commit
sha256sum "$BITSTREAM" 2>/dev/null              # FPGA bitstream identity

The hardware-access problem, named honestly

ASPLOS artifacts often need hardware an independent evaluator will not have. The honest pattern is a three-tier availability statement drafted at submission time:

1. Repeatable anywhere: simulator experiments and analysis scripts — full

configs and one command per figure.

1. Repeatable with named hardware: the exact platform requirements (board,

expander, CPU family), plus what to expect if the evaluator's part differs.

1. Not independently repeatable: results on lab-only or pre-production

hardware — say so, and provide either supervised access, raw logs with the analysis pipeline, or a scaled-down proxy. Silence here reads as concealment; a stated limitation reads as engineering.

Claim-preservation, not number-worship

State which conclusions should survive environmental drift and which are environment-specific: "the ordering of policies is stable across kernels 6.6-6.9; absolute runtimes are not." This single sentence pattern prevents the most common failed-reproduction dispute — an evaluator matching your ordering but not your absolute numbers and calling it a failure.

Timing across the ASPLOS cycle

• Before September 9: ledger current; capture script in the repo; availability

tiers drafted (they inform the paper's own text).

• Response window: the ledger is your defense when a reviewer doubts a number —

you can state the exact conditions instead of hand-waving.

• Major Revision: re-run under the captured original state where possible;

where the environment has drifted, disclose the drift in the change note.

• After acceptance: the ledger becomes the Artifact Appendix's dependency

section nearly verbatim; AE calendars for 2027 were 待核实 at pack-check time, so confirm dates when notified.

One command per figure

The internal gold standard that makes everything downstream cheap: every figure and table in the paper regenerates from a single committed command that reads raw results and emits the exact plot. It catches stale-figure bugs before submission, turns response-window questions into lookups, and becomes the Reproducible-badge run script with a rename. Institute it at the first result, when it costs minutes — retrofitting it at camera-ready costs days.

Trace and dataset provenance

Workload inputs decay independently of code. For each trace or dataset, record origin (public suite version, generated-by script + seed, or production source), preprocessing steps as scripts rather than prose, and a checksum of the exact bytes used. Production traces that cannot be released need a characterization (rate, skew, working-set curves) plus a matched synthetic generator committed to the repo — this is also the anonymity-safe form for submission, since a raw trace can identify its owner.

When numbers drift between submission and revision

The Major Revision window arrives months after the original runs, and environments drift. Protocol:

1. Re-run a sentinel subset (three representative experiments) under the

captured original state before starting revision work; if the sentinels reproduce, extend confidently.

1. If they do not, bisect the ledger — kernel, microcode, simulator commit —

until the moved variable is found; the ledger exists for exactly this moment.

1. Disclose in the change note which results were re-collected and under what

changed conditions, and re-state the claim-preservation sentence for the new environment. Silent regeneration of all numbers invites a reviewer to ask which version was real.

Output format

text
[Ledger coverage] platforms with complete rows: N/N · gaps listed
[Capture automation] script committed + outputs stored with data: Y/N
[Simulator pinning] commit + configs + region method + warm-up recorded: Y/N
[Availability tiers] anywhere / named-hardware / not-repeatable — each populated
[Claim preservation] drift-stable vs environment-specific conclusions stated: Y/N
[Badge readiness] which badges the current package could already earn