/autoresearch-agent

name: "autoresearch-agent"
description: "Autonomous experiment loop that optimizes any file by a measurable metric. Inspired by Karpathy's autoresearch. The agent edits a target file, runs a fixed evaluation, keeps improvements (git commit), discards failures (git reset), and loops indefinitely. Use when: user wants to optimize code speed, reduce bundle/image size, improve test pass rate, optimize prompts, improve content quality (headlines, copy, CTR), or run any measurable improvement loop. Requires: a target file, an evaluation command that outputs a metric, and a git repo."
license: MIT
metadata:
  version: 2.0.0
  author: Alireza Rezvani
  category: engineering
  updated: 2026-03-13

Autoresearch Agent

> You sleep. The agent experiments. You wake up to results.

Autonomous experiment loop inspired by [Karpathy's autoresearch](https://github.com/karpathy/autoresearch). The agent edits one file, runs a fixed evaluation, keeps improvements, discards failures, and loops indefinitely.

Not one guess — fifty measured attempts, compounding.

---

Slash Commands

| Command | What it does | |---------|-------------| | `/ar:setup` | Set up a new experiment interactively | | `/ar:run` | Run a single experiment iteration | | `/ar:loop` | Start autonomous loop with configurable interval (10m, 1h, daily, weekly, monthly) | | `/ar:ar-status` | Show dashboard and results | | `/ar:ar-resume` | Resume a paused experiment |

---

When This Skill Activates

Recognize these patterns from the user:

• "Make this faster / smaller / better"
• "Optimize [file] for [metric]"
• "Improve my [headlines / copy / prompts]"
• "Run experiments overnight"
• "I want to get [metric] from X to Y"
• Any request involving: optimize, benchmark, improve, experiment loop, autoresearch

If the user describes a target file + a way to measure success → this skill applies.

---

Setup

First Time — Create the Experiment

Run the setup script. The user decides where experiments live:

**Project-level** (inside repo, git-tracked, shareable with team):

python scripts/setup_experiment.py \
  --domain engineering \
  --name api-speed \
  --target src/api/search.py \
  --eval "pytest bench.py --tb=no -q" \
  --metric p50_ms \
  --direction lower \
  --scope project

**User-level** (personal, in `~/.autoresearch/`):

python scripts/setup_experiment.py \
  --domain marketing \
  --name medium-ctr \
  --target content/titles.md \
  --eval "python evaluate.py" \
  --metric ctr_score \
  --direction higher \
  --evaluator llm_judge_content \
  --scope user

The `--scope` flag determines where `.autoresearch/` lives:

• `project` (default) → `.autoresearch/` in the repo root. Experiment definitions are git-tracked. Results are gitignored.
• `user` → `~/.autoresearch/` in the home directory. Everything is personal.

What Setup Creates

.autoresearch/
├── config.yaml                        ← Global settings
├── .gitignore                         ← Ignores results.tsv, *.log
└── {domain}/{experiment-name}/
    ├── program.md                     ← Objectives, constraints, strategy
    ├── config.cfg                     ← Target, eval cmd, metric, direction
    ├── results.tsv                    ← Experiment log (gitignored)
    └── evaluate.py                    ← Evaluation script (if --evaluator used)

**results.tsv columns:** `commit | metric | status | description`

• `commit` — short git hash
• `metric` — float value or "N/A" for crashes
• `status` — keep | discard | crash
• `description` — what changed or why it crashed

Domains

| Domain | Use Cases | |--------|-----------| | `engineering` | Code speed, memory, bundle size, test pass rate, build time | | `marketing` | Headlines, social copy, email subjects, ad copy, engagement | | `content` | Article structure, SEO descriptions, readability, CTR | | `prompts` | System prompts, chatbot tone, agent instructions | | `custom` | Anything else with a measurable metric |

If `program.md` Already Exists

The user may have written their own `program.md`. If found in the experiment directory, read it. It overrides the template. Only ask for what's missing.

---

Agent Protocol

You are the loop. The scripts handle setup and evaluation — you handle the creative work.

Before Starting

1. Read `.autoresearch/{domain}/{name}/config.cfg` to get:

• `target` — the file you edit
• `evaluate_cmd` — the command that measures your changes
• `metric` — the metric name to look for in eval output
• `metric_direction` — "lower" or "higher" is better
• `time_budget_minutes` — max time per evaluation

2. Read `program.md` for strategy, constraints, and what you can/cannot change 3. Read `results.tsv` for experiment history (columns: commit, metric, status, description) 4. Checkout the experiment branch: `git checkout autoresearch/{domain}/{name}`

Each Iteration

1. Review results.tsv — what worked? What failed? What hasn't been tried? 2. Decide ONE change to the target file. One variable per experiment. 3. Edit the target file 4. Commit: `git add {target} && git commit -m "experiment: {description}"` 5. Evaluate: `python scripts/run_experiment.py --experiment {domain}/{name} --single` 6. Read the output — it prints KEEP, DISCARD, or CRASH with the metric value 7. Go to step 1

What the Script Handles (you don't)

• Running the eval command with timeout
• Parsing the metric from eval output
• Comparing to previous best
• Reverting the commit on failure (`git reset --hard HEAD~1`)
• Logging the result to results.tsv

Starting an Experiment

# Single iteration (the agent calls this repeatedly)
python scripts/run_experiment.py --experiment engineering/api-speed --single

# Dry run (test setup before starting)
python scripts/run_experiment.py --experiment engineering/api-speed --dry-run

Strategy Escalation

• Runs 1-5: Low-hanging fruit (obvious improvements, simple optimizations)
• Runs 6-15: Systematic ex