/experiment-pipeline

name: experiment-pipeline
description: "Guides structured 4-stage experiment execution with attempt budgets and gate conditions: Stage 1 initial implementation (reproduce baseline), Stage 2 hyperparameter tuning, Stage 3 proposed method validation, Stage 4 ablation study. Integrates with evo-memory (load prior strategies, trigger IVE/ESE) and experiment-craft (5-step diagnostic on failure). Use when: user has a planned experiment, needs to reproduce baselines, organize experiment workflow, or systematically validate a method. Do NOT use for debugging a specific experiment failure (use experiment-craft) or designing which experiments to run (use paper-planning)."
allowed-tools: "write_file edit_file read_file think_tool execute"
metadata:
  author: EvoScientist
  version: '1.0.0'
  tags: [core, experimentation, experiment-design]

Experiment Pipeline

A structured 4-stage framework for executing research experiments from initial implementation through ablation study, with attempt budgets and gate conditions that prevent wasted effort. This follows the Experiment Tree Search design from the EvoScientist paper, where the engineer agent iteratively generates executable code, runs experiments, and records structured execution results at each stage.

When to Use This Skill

• User has a planned experiment and needs to organize the execution workflow
• User wants to systematically validate a novel method against baselines
• User asks about experiment stages, attempt budgets, or when to move on
• User needs to reproduce baseline results before testing their method
• User mentions "experiment pipeline", "baseline first", "ablation study", "stage budget", "experiment execution"

The Pipeline Mindset

**Experiments fail for two reasons: wrong order and no stopping criteria.** Most researchers jump straight to testing their novel method without verifying their baseline setup, then wonder why results don't make sense. Others spend weeks tuning hyperparameters without a budget, hoping the next run will work.

The 4-stage pipeline solves both problems. It enforces a strict order (each stage validates assumptions the next stage depends on) and assigns attempt budgets (forcing systematic thinking over brute-force iteration).

Before Starting: Load Prior Knowledge

If coming from `research-ideation`, your research proposal (Step 7) provides the experiment plan — datasets, baselines, metrics, and ablation design — that maps directly to Stages 1-4 below.

Before entering the pipeline, load Experimentation Memory (M_E) from prior cycles:

1. Refer to the **evo-memory** skill → Read M_E at `/memory/experiment-memory.md` 2. Select the top-1 entry (k_E=1) most relevant to the current experiment domain by comparing each entry's Context and Category against the current problem 3. The selected strategy informs hyperparameter ranges (Stage 2), debugging approaches (Stages 1-3), and training configurations across all stages 4. If M_E doesn't exist yet (first cycle), skip this step and proceed — your results will seed M_E via ESE after pipeline completion

4-Stage Pipeline Overview

Each stage follows a **generate → execute → record → diagnose → revise** loop:

| Stage | Goal | Budget (N_E^s) | Gate Condition | |-------|------|--------|----------------| | 1. Initial Implementation | Get baseline code running and reproduce known results | ≤20 attempts | Metrics within 2% of reported values (or within reported variance) | | 2. Hyperparameter Tuning | Optimize config for your setup | ≤12 attempts | Stable config, variance < 5% across 3 runs | | 3. Proposed Method | Implement & validate novel method | ≤12 attempts | Outperforms tuned baseline on primary metric, consistent across 3 runs | | 4. Ablation Study | Prove each component's contribution | ≤18 attempts | All claims evidenced with controlled experiments |

Each stage saves artifacts to `/experiments/stageN_name/`.

The Stage Loop

Within every stage, repeat this cycle for each attempt:

1. **Generate**: Form a hypothesis or plan for this attempt. What specifically will you try? What do you expect to happen? 2. **Execute**: Run the experiment. Record exact configuration, code changes, and runtime. 3. **Record**: Log results immediately using the stage log template. Include both metrics and observations. 4. **Diagnose**: Compare results to expectations. If they match, assess the gate condition. If they don't, load `experiment-craft` for the 5-step diagnostic flow. 5. **Revise**: Based on diagnosis, either advance to the next stage (gate met) or plan the next attempt (gate not met).

Stage 1: Initial Implementation

**Goal**: Find or generate executable baseline code and verify it reproduces published results. This stage corresponds to the paper's "initial implementation" — the engineer agent searches for working code, runs it, and records structured execution results.

**Why this matters**: If you can't get the baseline running and reproducing known results, every subsequent comparison is meaningless. Initial implementation validates your data pipeline, evaluation code, training infrastructure, and understanding of prior work.

**Budget**: ≤20 attempts (N_E^1=20). Baselines can be tricky — missing details in papers, version mismatches, unreported preprocessing steps. 20 attempts gives enough room to debug without allowing infinite tinkering.

**Gate**: Primary metrics within 2% of reported values (or within the reported variance if provided).

**Process**: 1. Find the original baseline code (official repo, re-implementations, or write from paper description) 2. Get the code running in your environment — resolve dependencies, fix compatibility issues 3. Match the exact training configuration from the paper (dataset splits, preprocessing, hyperparameters) 4. Run and compare metrics. If off by >2%, diagnose the gap 5. Common pitfalls: different random seeds, different data splits, unreported data augmentation, framework version differences