/asplos-reproducibility

name: asplos-reproducibility
description: Use when hardening an ASPLOS paper's results for independent repetition — pinning simulator versions and configs, recording kernel/firmware/BIOS state, packaging FPGA bitstreams and RTL, documenting hardware dependencies an evaluator may lack, and writing availability statements that match what the ACM badges will later require.

ASPLOS Reproducibility

Systems results decay fast: a kernel update, a microcode revision, or a silently changed simulator default can move numbers by more than the paper's claimed margin. Reproducibility work at ASPLOS is therefore **state capture** — recording the full machine, model, and toolchain state behind every figure — done while the experiments run, not reconstructed at camera-ready time. It also front-loads artifact evaluation: the badge criteria (`asplos-artifact-evaluation`) are exactly a demand that this state capture exists and works.

The state ledger

Maintain one ledger row per experimental platform, committed alongside results:

| Layer | Capture | Why it moves numbers | |---|---|---| | Silicon | CPU model + stepping, memory config/topology, device (e.g. CXL expander) firmware | Steppings differ in errata and prefetch behavior | | Firmware/BIOS | Microcode revision; SMT, turbo, prefetcher, C-state, NUMA settings | Any one knob can swamp a 10% effect | | OS | Kernel version + full config, relevant sysctls, mitigations state | Speculation mitigations alone shift syscall-heavy results | | Toolchain | Compiler + flags, libraries, runtime versions | -O level and allocator choice are classic silent variables | | Simulator | Exact commit, all config files, region/checkpoint method, warm-up length | Defaults change across releases without notice | | FPGA | Board, toolchain version, constraints, bitstream hash, achieved clock | Re-synthesis at a different clock is a different experiment | | Workloads | Suite versions, input sets, trace provenance and preprocessing | "SPEC" without input class is unrepeatable | | Randomness | Seeds for any stochastic component + run counts | Needed for the dispersion numbers to mean anything |

Scripted capture beats remembered capture

Run at the start of every measurement session; store output next to the data:

#!/bin/sh
# state-capture.sh — commit this file and its output with each result set
uname -a; cat /proc/cmdline
grep -m1 'model name' /proc/cpuinfo; grep microcode /proc/cpuinfo | sort -u
cat /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | sort -u
cat /sys/devices/system/cpu/smt/control 2>/dev/null
numactl --hardware 2>/dev/null | head -5
cc --version | head -1
git -C "$SIM_DIR" rev-parse HEAD 2>/dev/null   # simulator commit
sha256sum "$BITSTREAM" 2>/dev/null              # FPGA bitstream identity

The hardware-access problem, named honestly

ASPLOS artifacts often need hardware an independent evaluator will not have. The honest pattern is a **three-tier availability statement** drafted at submission time:

1. **Repeatable anywhere:** simulator experiments and analysis scripts — full configs and one command per figure. 2. **Repeatable with named hardware:** the exact platform requirements (board, expander, CPU family), plus what to expect if the evaluator's part differs. 3. **Not independently repeatable:** results on lab-only or pre-production hardware — say so, and provide either supervised access, raw logs with the analysis pipeline, or a scaled-down proxy. Silence here reads as concealment; a stated limitation reads as engineering.

Claim-preservation, not number-worship

State which conclusions should survive environmental drift and which are environment-specific: "the ordering of policies is stable across kernels 6.6-6.9; absolute runtimes are not." This single sentence pattern prevents the most common failed-reproduction dispute — an evaluator matching your ordering but not your absolute numbers and calling it a failure.

Timing across the ASPLOS cycle

• **Before September 9:** ledger current; capture script in the repo; availability

tiers drafted (they inform the paper's own text).

• **Response window:** the ledger is your defense when a reviewer doubts a number —

you can state the exact conditions instead of hand-waving.

• **Major Revision:** re-run under the *captured* original state where possible;

where the environment has drifted, disclose the drift in the change note.

• **After acceptance:** the ledger becomes the Artifact Appendix's dependency

section nearly verbatim; AE calendars for 2027 were 待核实 at pack-check time, so confirm dates when notified.

One command per figure

The internal gold standard that makes everything downstream cheap: every figure and table in the paper regenerates from a single committed command that reads raw results and emits the exact plot. It catches stale-figure bugs before submission, turns response-window questions into lookups, and becomes the Reproducible-badge run script with a rename. Institute it at the first result, when it costs minutes — retrofitting it at camera-ready costs days.

Trace and dataset provenance

Workload inputs decay independently of code. For each trace or dataset, record origin (public suite version, generated-by script + seed, or production source), preprocessing steps as scripts rather than prose, and a checksum of the exact bytes used. Production traces that cannot be released need a characterization (rate, skew, working-set curves) plus a matched synthetic generator committed to the repo — this is also the anonymity-safe form for submission, since a raw trace can identify its owner.

When numbers drift between submission and revision

The Major Revision window arrives months after the original runs, and environments drift. Protocol:

1. Re-run a **sentinel subset** (three representative experiments) under the captured original state before starting revision work; if the sentinels reproduce, extend confidently.