/help

description: "Explain autoresearch plugin and available commands"

Autoresearch Plugin Help

Explain the following to the user:

What is autoresearch?

A Claude Code plugin inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch) — an autonomous research loop where Claude acts as a researcher, continuously editing one artifact, running a scorer, logging results, and iterating overnight without human intervention.

Unlike the original (which is hardwired to ML training), this plugin is **domain-agnostic**: you supply the editable artifact, the scorer, and the optimization direction, so the loop works on any problem that reduces to *"edit something, run a command that prints one number, keep the change if the number improved."* ML training is just one configuration of it.

**Core idea:**

• Point Claude at one artifact it may edit and one scorer command that prints a number
• Let it experiment autonomously — make a change, score it, keep or discard, repeat
• Wake up in the morning to a log of experiments and (hopefully) a better score

**How the loop runs (this plugin's mechanism):**

Rather than an external shell loop, this plugin uses a Claude Code **Stop hook**. Every time Claude tries to end its turn, the hook intercepts the exit and re-injects the same research prompt — so Claude keeps experimenting in one continuous session. Claude sees its previous work in git history and `results.tsv`, building incrementally toward a better score. The hook stops the loop when a configured bound is hit (max experiments, wall-clock budget, or a completion promise).

This is the same spirit as the "ralph-loop" idea behind the original project (re-feeding a prompt until done), adapted to Claude Code's hook system instead of a `while` loop over a CLI.

Requirements

• A git repository (the loop runs in a dedicated git **worktree** so your main checkout, current branch, and even a dirty tree are never touched)
• At least one bound: `--max-experiments` and/or `--max-wall-clock`
• An evaluator: a `--score-cmd` (prints a number as its **last** stdout line) and/or a `--check-cmd` (objective pass/fail gate, exit 0 = pass)
• Whatever runtime the evaluator needs (interpreter, data, GPU, ...) — that is its concern, not the plugin's

Before a long run: raise the Stop-hook block cap

Claude Code force-ends the turn after a Stop hook blocks `CLAUDE_CODE_STOP_HOOK_BLOCK_CAP` times in a row (**default 8**). This loop re-blocks once per experiment, so any run that wants more than ~8 experiments will hit that ceiling and stop early unless you raise it.

Autoresearch already enforces its own bound (experiments / wall-clock / completion promise) inside the same Stop hook, so the generic cap is redundant here — disable it. Add to `.claude/settings.json` (or `~/.claude/settings.json`):

{ "env": { "CLAUDE_CODE_STOP_HOOK_BLOCK_CAP": "0" } }

`"0"` disables the cap; or set a number comfortably above your experiment count. The env var is read at **session start**, so set it and restart Claude Code *before* the run — a plugin cannot set it for you, and it does not take effect mid-session. `/autoresearch:start` prints a reminder when your configured bound exceeds the active cap.

Commands

/autoresearch:start <goal> [overrides]

Give it a plain-language goal — `/autoresearch:start make X faster` — and it inspects the repo to infer the full contract, asking only when the artifact or evaluator is genuinely ambiguous. Pass any flag below to pin it as an override.

**Inferred (or overridden) contract:**

• `--edit <glob|path>` — the ONLY artifact the agent may modify
• An **evaluator** — at least one of:
• `--score-cmd '<shell>'` — numeric scorer; LAST stdout line is a single number (needs `--direction`)
• `--check-cmd '<shell>'` — objective gate; exit 0 = pass. Use alone to "iterate until it passes", or with `--score-cmd` as a hard filter on top of optimization.
• `--rubric '<criteria>'` — criteria an LLM judge panel applies when a plateau escalates to a tournament. Must be anchored by `--score-cmd` or `--check-cmd` (a judge-only loop reward-hacks).
• `--direction min|max` — lower or higher score is better (only with `--score-cmd`)
• `--prompt` / `--objective` — the goal and its measurable restatement (auto-filled)

Plus at least one bound (`--max-experiments` or `--max-wall-clock`); the inference fills sensible defaults.

**Options:**

• `TAG` — branch suffix (e.g. `mar16`); defaults to today's date
• `--max-experiments <n>` — stop after N experiments
• `--max-wall-clock <duration>` — stop after a wall-clock budget (e.g. `8h`, `480m`, `30s`)
• `--readonly <path>` — protect a path from edits (repeatable)
• `--trial-timeout <duration>` — hard time limit per scorer run (default `600`s; plain numbers are seconds, unlike `--max-wall-clock`)
• `--precheck '<shell>'` — a precondition that must exit 0 before the loop starts (repeatable)
• `--completion-promise <text>` — Claude outputs `<promise>TEXT</promise>` to signal done
• `--force` — replace an existing active loop state file (use after `/autoresearch:cancel` if needed)
• `-h`, `--help` — show usage and exit

**Examples:**

# Data cleaning — maximize F1
/autoresearch:start clean1 --prompt 'improve the cleaning rules' \
  --objective 'maximize F1, precision >= 0.90' \
  --edit clean.py --readonly score.sh --score-cmd 'bash score.sh' \
  --direction max --max-experiments 20

# Prompt optimization — maximize eval accuracy
/autoresearch:start --prompt 'raise accuracy on the eval set' \
  --objective 'maximize accuracy' --edit prompt.txt \
  --score-cmd 'python eval_prompt.py --set val.jsonl' \
  --direction max --max-experiments 30

# ML training parity — reproduce the original behavior
/autoresearch:start --prompt 'lower validation bits-per-byte' \
  --objective 'minimize val_bpb' --edit train.py \
  --readonly prepare.py --readonly evaluate_bpb \
  --score-cmd 'timeout 600 uv run train.py >run.log 2>&1; grep "^val_bpb:"