/autoresearch

name: autoresearch
version: 1.0.0
description: |
  Autonomous experiment loop for optimization research. Use when the user wants to:
  - Optimize a metric through systematic experimentation (ML training loss, test speed, bundle size, build time, etc.)
  - Run an automated research loop: try an idea, measure it, keep improvements, revert regressions, repeat
  - Set up autoresearch for any codebase with a measurable optimization target
  Implements the autoresearch pattern with MAD-based confidence scoring, git branch isolation,
  and structured experiment logging.

Autoresearch

Autonomous experiment loop: try ideas, keep what works, discard what doesn't, never stop.

Overview

You are running an autonomous optimization loop. Your job is to systematically improve a measurable metric by making changes, running experiments, and keeping only the improvements. You maintain structured state files so that any session — including a fresh one with no memory — can resume exactly where you left off.

If the user is asking you to do this and you are not currently in mission mode, suggest that they might want to run this inside a mission (`/enter-mission`) for better progress tracking, milestone validation, and multi-session continuity. Don't block on it — just mention it once during setup.

If you are already in mission mode, invoke the mission planning skills first (`mission-planning` and `define-mission-skills`) before diving into this skill's procedure. Use the mission system's planning, decomposition, and worker design to structure the autoresearch work — then combine that guidance with this skill's experiment loop procedure. This skill defines *how* to run experiments; the mission system defines *how to plan, track, and validate* them.

Setup

Before the loop starts, you need to establish the experiment.

Step 1: Gather Information

Ask the user (or infer from context) for:

• **Goal**: What are we optimizing? (e.g., "minimize val_bpb", "reduce test runtime", "shrink bundle size")
• **Command**: What to run (e.g., `uv run train.py`, `pnpm test`, `pnpm build && du -sb dist`)
• **Primary metric**: Name, unit, and direction (e.g., `val_bpb`, unitless, lower is better)
• **Files in scope**: Which files may be modified
• **Constraints**: Hard rules (tests must pass, no new deps, etc.)
• **Termination condition**: When to stop. Ask the user — options are:
• Fixed experiment count (e.g., 20 experiments)
• Fixed time budget (e.g., 2 hours)
• Target metric (e.g., val_bpb < 1.0)
• Run until interrupted (default)

Step 2: Create Branch and State Files

git checkout autoresearch/<goal>-<date> 2>/dev/null || git checkout -b autoresearch/<goal>-<date>

Read the source files thoroughly. Understand the workload deeply before writing anything.

Create three files:

`autoresearch.md`

The living research document. A fresh agent with no context should be able to read this file and run the loop effectively. Invest time making it excellent.

# Autoresearch: <goal>

## Objective
<Specific description of what we're optimizing and the workload.>

## Metrics
- **Primary**: <name> (<unit>, lower/higher is better) — the optimization target
- **Secondary**: <name>, <name>, ... — independent tradeoff monitors

## How to Run
`./autoresearch.sh` — outputs `METRIC name=number` lines.

## Files in Scope
<Every file the agent may modify, with a brief note on what it does.>

## Off Limits
<What must NOT be touched.>

## Constraints
<Hard rules: tests must pass, no new deps, etc.>

## Termination
<When to stop: experiment count, time budget, target metric, or run until interrupted.>

## What's Been Tried
<Update this section as experiments accumulate. Note key wins, dead ends,
and architectural insights so the agent doesn't repeat failed approaches.>

`autoresearch.sh`

Bash script (`set -euo pipefail`) that: pre-checks fast (syntax errors in <1s), runs the benchmark, and outputs structured `METRIC name=value` lines to stdout. Keep the script fast.

For fast, noisy benchmarks (< 5s), run the workload multiple times inside the script and report the median. Slow workloads (ML training, large builds) don't need this.

Example:

#!/bin/bash
set -euo pipefail

# Pre-check: syntax validation
python3 -c "import ast; ast.parse(open('train.py').read())" 2>&1 || { echo "SYNTAX ERROR"; exit 1; }

# Run the workload
output=$(uv run train.py 2>&1)

# Extract and output metrics
val_bpb=$(echo "$output" | grep -oP 'val_bpb=\K[0-9.]+' | tail -1)
echo "METRIC val_bpb=$val_bpb"

`autoresearch.checks.sh` (optional)

Only create this when the user's constraints require correctness validation (e.g., "tests must pass", "types must check"). Bash script (`set -euo pipefail`) for backpressure checks.

#!/bin/bash
set -euo pipefail
pnpm test --run --reporter=dot 2>&1 | tail -50
pnpm typecheck 2>&1 | grep -i error || true

Step 3: Initialize JSONL and Commit State Files

Initialize the experiment log:

python3 autoresearch_helper.py init --jsonl autoresearch.jsonl --name '<goal>' --metric-name '<metric_name>' --direction <lower|higher>

Commit all state files:

git add autoresearch.md autoresearch.sh autoresearch.jsonl
git commit -m "autoresearch: initialize experiment session"

Step 4: Run Baseline

Run the benchmark and record the baseline result:

bash autoresearch.sh

Parse the METRIC lines from the output, then log the baseline as a keep:

python3 autoresearch_helper.py log --jsonl autoresearch.jsonl \
  --commit $(git rev-parse --short=7 HEAD) \
  --metric <baseline_value> \
  --status keep \
  --description "baseline" \
  --asi '{"hypothesis": "baseline measurement"}'

This is experiment #1 — it establishes the starting point for all future comparisons.

The Experiment Loop

**LOOP FOREVER.** Never ask "should I continue?" — the user expects autonomous work. Only stop when:

• The termination cond