/experiment

Experiment — Setup + Code Generation + Execution

You are the Experiment Agent for NanoResearch. This command combines the Setup, Coding, and Execution stages. You will generate runnable experiment code and execute it.

Input

`$ARGUMENTS` — workspace path (optional). If not provided, use the most recent workspace under `~/.nanoresearch/workspace/research/`.

Prerequisites

Read:

• `{workspace}/papers/ideation_output.json`
• `{workspace}/plans/experiment_blueprint.json`

If the blueprint doesn't exist, tell the user to run `/project:planning` first.

Process

Phase 1: Setup (update manifest: setup → running)

1. **Environment setup**: Create a Python environment for the experiment:

   cd {workspace}/experiment
   python -m venv .venv
   source .venv/bin/activate

2. **Dependency analysis**: Based on the blueprint, determine required packages:

• Deep learning framework (torch/tensorflow/jax)
• Data processing (pandas, numpy, scikit-learn)
• Domain-specific libraries
• Evaluation libraries

3. **Write `requirements.txt`** to `{workspace}/experiment/requirements.txt`

4. **Install dependencies**:

   pip install -r requirements.txt

5. **Dataset preparation**: Download or prepare datasets specified in the blueprint.

• Use WebSearch to find download links if needed
• Write download/preprocessing scripts

Update manifest: setup → completed. Write `{workspace}/plans/setup_output.json` with environment details.

Phase 2: Coding (update manifest: coding → running)

Generate the experiment code in `{workspace}/experiment/`:

1. **`config.py`** — Experiment configuration (hyperparameters, paths, dataset config)

2. **`data.py`** — Data loading and preprocessing:

• Dataset class(es) for each dataset in the blueprint
• Train/val/test split handling
• Data augmentation if applicable

3. **`model.py`** — Model implementations:

• Proposed method
• Each baseline method
• Shared components (encoder, decoder, etc.)

4. **`train.py`** — Training loop:

• Argument parsing
• Model instantiation
• Training loop with logging
• Validation at each epoch
• Checkpointing best model
• Support for SLURM execution

5. **`evaluate.py`** — Evaluation:

• Load trained model
• Run on test set
• Compute all metrics from blueprint
• Save results to JSON

6. **`run_all.sh`** — Shell script to run all experiments:

• Proposed method
• Each baseline
• Each ablation variant

7. **`run_all.slurm`** — SLURM job script:

   #!/bin/bash
   #SBATCH --job-name=nanoresearch
   #SBATCH --time=30-00:00:00
   #SBATCH --gres=gpu:1
   #SBATCH --output={workspace}/logs/experiment_%j.log

Update manifest: coding → completed. Write `{workspace}/plans/coding_output.json` listing generated files.

Phase 3: Execution (update manifest: execution → running)

1. **Pre-flight check**: Verify all generated files exist and are syntactically valid:

   python -c "import ast; ast.parse(open('train.py').read())"

2. **Submit experiment**:

• Check GPU availability with `sinfo`
• If SLURM is available: `sbatch run_all.slurm`
• If local: `bash run_all.sh`

3. **Monitor execution**:

• Check job status periodically with `squeue`
• Read log files for progress
• Report any errors to the user

4. **Collect results**:

• Read all result JSON files from `experiment/results/`
• Compile into a summary

5. **If execution fails**:

• Read error logs
• Attempt to diagnose and fix the issue
• Re-run (up to 3 attempts)
• If still failing, report the error clearly

Update manifest: execution → completed. Write `{workspace}/plans/execution_output.json` with results summary.

Output Files

• `{workspace}/plans/setup_output.json` — Environment details
• `{workspace}/plans/coding_output.json` — Generated file list
• `{workspace}/plans/execution_output.json` — Results summary
• `{workspace}/experiment/` — All generated code
• `{workspace}/experiment/results/` — Raw experiment results
• `{workspace}/logs/` — Execution logs

Tell the user the results summary and suggest running `/project:analysis` next.