/ase-experiments

name: ase-experiments
description: Use when designing or auditing the evaluation of an ASE (IEEE/ACM Automated Software Engineering) paper, covering real subject systems, fair runnable tool baselines, task-matched effectiveness metrics, ablations that isolate a learned component, oracle and correctness validation, contamination-aware LLM handling, and provenance for mining.

ASE Experiments

Match the evidence to the automation's claim. ASE evaluations are judged on whether a **tool or technique actually does what it claims on real subjects**, compared **fairly** against the closest runnable automation. This is the axis reviewers weight most, and the one that most often becomes a Revision criterion.

Start from the claim shape

Different automations demand different evidence:

| Automation claim | Evidence that matches | Common failure | |---|---|---| | Detection (bugs, smells, vulnerabilities) | Precision/recall/F on real defects with a defined ground truth | Synthetic-only defects; unclear ground truth | | Generation / synthesis (tests, code, patches) | Validity of the produced artifact (compiles, passes, holds the property) | Similarity-to-reference proxy instead of validity | | Repair | Verified behavior change: re-run + oracle; assertion/spec preservation | "Plausible patch" without an overfitting check | | Localization / ranking | Rank-based effectiveness on real faults vs. alternatives | Cherry-picked programs; one metric only | | Scalability / performance | Real-system sizes, wall-clock with a fair config | Toy inputs; unequal baseline budget |

Real subject systems

• Use **real software** — open-source projects, real bug/defect datasets, real CI logs — not toy

programs you constructed to make the tool look good.

• Report subject **provenance**: names, versions/commit SHAs, sizes, and the extraction date.

Reviewers reproduce from this.

• Justify subject **selection** and disclose exclusions; self-selected subjects are the classic

external-validity threat.

Fair, runnable tool baselines

• Compare against the **closest runnable automation**, configured at an **equal, documented budget**

(time, iterations, tuning, seeds). ASE reviewers routinely rerun or scrutinize baselines.

• Pin baseline **versions/commits** and note reimplementation vs. original.
• If no tool baseline exists, construct a defensible non-trivial baseline (a static rewrite, a

random or heuristic variant) rather than comparing only to "nothing."

Ablations that isolate the automation

If a learned or LLM component is involved, run an **ablation** that removes it and keeps the rest, so the marginal value of the *design* is visible. This is what defeats the "the model did it, not your technique" objection and keeps the paper ASE-shaped rather than ML-shaped.

Oracles and correctness

• State the **oracle** explicitly: how do you know a generated test is meaningful, or a repair is

correct? Re-execution, differential testing, formal checks, or human audit — name it.

• For repair/synthesis, guard against **overfitting** to the evaluation oracle (e.g., patches that

pass the given tests but break behavior): report a held-out or manual correctness check.

Statistics and effect sizes

• Report **effect sizes and dispersion** (confidence intervals, non-parametric tests where

appropriate), not just point estimates or a single accuracy number.

• For randomized techniques (search-based, sampling, LLM temperature > 0), report **repeated runs**

with variance and fix/seed the randomness for the artifact.

Contamination-aware LLM handling

• Record **model identifiers and dates**; a model updated between runs invalidates comparisons.
• Consider **training-data contamination**: benchmarks the model may have seen inflate results —

report on held-out or post-cutoff subjects where feasible, and say so.

• **Cache raw model outputs** so the artifact reproduces rather than re-samples a live API.

Mining and dataset provenance

• Pin repository SHAs, the corpus extraction date, query/filter criteria, and any labeling

protocol with inter-rater agreement for manually coded data.

• Version the dataset and describe how to regenerate it; a package that needs live scraping

re-samples a moving target.

Evaluation audit checklist

[Claim-evidence] each claim -> a matching metric on real subjects (not a proxy)
[Subjects] real, provenance-pinned, selection justified, exclusions disclosed
[Baselines] closest runnable tool, version pinned, equal documented budget
[Ablation] learned/LLM component isolated; marginal value of the design shown
[Oracle] correctness defined; overfitting-to-oracle checked
[Stats] effect sizes + dispersion; repeated runs for randomized methods
[LLM] model IDs/dates recorded; contamination considered; outputs cached
[Repro] provenance pinned; dataset/tool versioned for the artifact

Output format

[Automation claim] detection / generation / repair / localization / scalability
[Evidence match] metric(s) that fit the claim, on real subjects
[Baseline fairness] closest tool, budget parity, versions
[Ablation + oracle] learned-component ablation present; correctness oracle stated
[Threats] subject selection / oracle validity / baseline fairness / contamination — bounded how?