/fortify

name: fortify
description: Systematic ablation study runner. After research:run finds improvements, fortify identifies component candidates from git diff + diary, creates isolated git worktrees per ablation (main repo never modified), runs metric+guard in each worktree, ranks component importance, and optionally generates reviewer Q&A calibrated to a target venue.
argument-hint: '[<run-id>|<program.md>] [--venue <CVPR|NeurIPS|ICML|workshop>] [--max-ablations <N>] [--runs <N>] [--skip-run] [--keep "<items>"]'
effort: high
allowed-tools: Read, Write, Bash, Agent, TaskCreate, TaskUpdate, AskUserQuestion
disable-model-invocation: true

<objective>

Ablation study runner — after `/research:run` finds improvements, fortify identifies which components contributed, generates ablation variants (remove one component at a time), runs each in **isolated git worktrees** (main repo never modified), ranks component importance, optionally generates reviewer Q&A calibrated to venue.

NOT for: initial optimization loop (use `/research:run`); methodology validation (use `/research:judge`); paper-vs-code consistency (use `/research:verify`); hypothesis generation (use `research:scientist` directly). Fortify runs ablation studies on completed runs only.

</objective>

<constants>

MAX_ABLATION_CANDIDATES:  8 (ceiling — scientist produces 3–8; --max-ablations caps further)
METRIC_TIMEOUT_MS:        config.metric_timeout_sec × 1000 (same effective metric limit as plan/judge/run; outer tool budget adds cleanup time)
GUARD_TIMEOUT_MS:         360000
GIT_OP_TIMEOUT_MS:        15000
METRIC_RUNS_DEFAULT:      3 (metric runs per variant; --runs <N> overrides, 1–10; 1 = no σ, threshold = noise floor alone)
SANITY_DIVERGENCE_PCT:    2.0 (full-variant mean vs best_metric tolerance, widened to 2σ_full; beyond it fortify aborts)
NOISE_FLOOR_PCT:          2.0 (% of |full mean|; an ablation counts as an effect only beyond max(noise floor, 2σΔ))
IMPORTANCE_CLASS_CRITICAL: 50.0 (% of full metric lost)
IMPORTANCE_CLASS_SIGNIFICANT: 10.0
IMPORTANCE_HARMFUL_PCT:   -5.0 (importance below this = Potentially Harmful; [-5.0, -threshold) = NEUTRAL)
FORTIFY_DIR_BASE:         .experiments
STATE_DIR_BASE:           .experiments/state
METRIC_CMD_SOURCE:        state.json .config.metric_cmd (campaign's real metric — no fail-open default; env METRIC_CMD overrides)
GUARD_CMD_SOURCE:         state.json .config.guard_cmd (campaign's real guard — no fail-open default; env GUARD_CMD overrides)

</constants>

**Environment overrides** — set these before invoking the skill to override per-variant defaults:

• `METRIC_CMD` — command run inside each variant worktree to measure the ablation metric (override; default sourced from source-run `state.json` `.config.metric_cmd`)
• `GUARD_CMD` — command run inside each variant worktree to detect regressions (override; default sourced from source-run `state.json` `.config.guard_cmd`)
• `STATE_DIR_BASE` — base directory for source-run state lookups (default: `.experiments/state`)
• `FORTIFY_DIR_BASE` — base directory for per-run fortify artifacts (default: `.experiments`)
• `SANITY_DIVERGENCE_PCT` / `NOISE_FLOOR_PCT` — full-variant reproduction tolerance and ablation noise floor, in percent (defaults above)

The constants block defaults above are YAML-only — bash blocks read environment variables (with `${VAR:-default}` fallback) and never source values directly from YAML.

<compaction>

• Key boundaries: end of F2 — ablation candidates identified by scientist; end of F4 — all ablation variant worktrees completed.
• Preserve at F2: FORTIFY_DIR (TMPDIR key), RUN_ID (TMPDIR key), ablation-candidates.jsonl path, best_metric from source run.
• Preserve at F4: FORTIFY_DIR, RUN_ID, results.jsonl and metric-runs.jsonl paths, fortify-runs (TMPDIR key) — ready for F5 importance ranking.
• 4d inner loop is resume-safe too: the run count per variant comes from `metric-runs.jsonl`, never memory, so a compaction mid-variant resumes at the next run.
• F4 loop is resume-safe: 4a-init's resume guard skips variants already terminal (non-timeout) in results.jsonl, so a mid-loop compaction resumes at the first pending variant — never re-runs a completed ablation; a timed-out variant still retries.
• Clear at F1 start (stale prior run) and at start of F8 terminal summary.

</compaction>

<workflow>

<!-- Agent resolution: see _RESEARCH_SHARED/agent-resolution.md -->

<!-- ../../assets/images/fortify-arch.svg diagrams the runs, gates and parallel fan-out. Documentation only, never loaded — update it in the same commit as any change to step order, gate placement, or agent fan-out. -->

Agent Resolution

**Agent resolution**: load and follow the protocol below. Agent this skill dispatches: `research:scientist` (same plugin — always present while this skill runs, so no fallback row).

# loads: compaction-contract.md
PLUGIN_ROOT=$(python "${CLAUDE_PLUGIN_ROOT}/bin/resolve_plugin_root.py" --plugin-name research) || exit $?; export PLUGIN_ROOT
export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
python "$PLUGIN_ROOT/bin/resolve_shared.py" --persist research-shared --emit agent-resolution.md || exit 1  # timeout: 5000 — cold resolve; every later site reads the research-shared sentinel instead of re-running python

CRITICAL: Worktree-based isolation

**Do NOT use `git checkout -b <branch>` for ablations** — dirties main working tree, corrupts concurrent tool calls. Each ablation gets own git worktree under `$FORTIFY_DIR/worktrees/<variant>`, created from `best_commit`. Main working tree NEVER modified. Cleanup: `git worktree remove --force` per variant; `git worktree prune` on interrupt.

Fortify Mode (Steps F1–F8)

**Task tracking**: create tasks for F1, F2, F3, F4, F5, F6, F7, F8 at start, all in the same response as the first real tool call. Every later `TaskUpdate` rides with the next real tool call — never a bookkeeping-only turn; the one standalone call is a `completed` right before a long repo