/experiment-reproducibility

name: experiment-reproducibility
description: "Make a thesis's computational experiments reproducible by the examiner and by the student three months later: pinned environments, seeded randomness, a single config per run, a run manifest that records code version, data version, parameters, hardware, and timings, results written to versioned files that figures and tables are generated from, and a one-command rerun. Use when setting up experiments, before the first real run, or when a result cannot be regenerated."
argument-hint: "(optional) the experiment directory, the framework, and where results should go"

A result that cannot be regenerated from a command is an anecdote. Reproducibility is cheap at the start and impossible at the end, so this is set up before the first run that will appear in the thesis.

Layout

experiments/
  env/          environment.yml or requirements.txt with exact pins; Dockerfile if system deps matter
  configs/      one YAML per experiment: exp01-baseline.yaml, exp02-ours.yaml
  data/         raw/ (never modified), processed/ (generated by scripts), DATA.md (source, licence, version, checksum)
  src/          the code; run.py takes --config and --seed
  runs/         one directory per run: <exp>-<seed>-<timestamp>/ with manifest.json, metrics.json, log.txt
  results/      aggregated tables (CSV) generated from runs/ by aggregate.py
  figures/      generated from results/ by figure scripts (research-figures)
  Makefile      or justfile: `make exp01` runs all seeds; `make results` aggregates; `make figures`

Everything under `runs/`, `results/`, and `figures/` is generated; regenerate rather than edit. Commit `results/` and `figures/` (they are small) so the thesis builds from a clean clone; do not commit `runs/` unless small (use DVC or a release asset otherwise).

Environment

• Python: `conda env export --no-builds` or `pip freeze` into `env/`, with the Python version. Better: `pyproject.toml` with a lockfile (`uv lock`, `poetry lock`).
• Record the CUDA/driver version and the exact framework build when GPUs are involved; results differ across them.
• A `Dockerfile` when anything outside the language environment matters (system libraries, a database).
• The manifest records `python --version`, `pip freeze` hash, `nvidia-smi` output, and `uname -a` at run time regardless.

Determinism

• One `--seed` argument that seeds every source: `random`, `numpy`, `torch` (and `torch.cuda.manual_seed_all`), `tf`, data shuffling, and worker init. Use a helper `seed_everything(seed)` called first.
• Set deterministic flags where the framework offers them (`torch.use_deterministic_algorithms(True)`, `CUBLAS_WORKSPACE_CONFIG`) and record whether they were on; some ops stay nondeterministic on GPU and the thesis should say so.
• Run every experiment with at least three seeds (five or ten when cheap); report mean and SD or CI (`statistics-advisor`). A single-seed number is not a result.

Configuration

• All parameters in the config file; nothing hard-coded, nothing from the command line except `--config` and `--seed` (and `--out`). The config is copied into the run directory unchanged.
• Named experiments, not edited-in-place configs: `exp03-ours-lr0.001.yaml` rather than changing `exp02` and losing what produced the earlier result.

The run manifest

`runs/<run>/manifest.json`, written at start and completed at end:

{
  "experiment": "exp02-ours", "seed": 1, "started": "2026-09-21T10:00:00Z", "finished": "...",
  "git": { "commit": "a1b2c3d", "dirty": false, "branch": "main" },
  "config_sha256": "...", "data_version": "v3 (sha256 ...)",
  "env": { "python": "3.12.4", "packages_sha256": "...", "cuda": "12.4", "gpu": "A100 40GB", "host": "..." },
  "duration_s": 1832.4, "status": "ok"
}

Refuse to start a run that will appear in the thesis from a dirty working tree (`git status --porcelain` non-empty): commit first, so the commit hash means something. Log the refusal.

Data

`data/DATA.md`: source URL, download date, licence, version, checksum of the raw archive, the preprocessing script and its parameters, the resulting row counts and split sizes. Splits are made once with a seed and saved as index files; every run reads the same splits.

Aggregation and figures

`aggregate.py` reads every `runs/*/metrics.json`, joins with the manifests, and writes `results/<table>.csv` with one row per (experiment, seed) and a summary with mean, SD, n. Figure scripts read only `results/`. The thesis `\input`s tables generated from `results/` (`benchmark-reporting`), so a rerun updates the document.

One command

`make all` (or `just all`): environment check, all experiments for all seeds, aggregate, figures, tables. Document it in `experiments/README.md` with the expected total runtime and hardware. The examiner, or a future student, runs it and gets the thesis's numbers.

Checklist before a result enters the thesis

• Run from a clean commit, manifest present, seeds ≥ 3.
• Config committed, data version recorded, environment pinned.
• Table and figure generated from `results/`, not typed.
• The commit hash cited in the thesis appendix.