/autoresearch

name: autoresearch
description: Autonomous goal-directed iteration loop that continuously improves prompts, templates, configs, or code. Two evaluation modes — deterministic (eval.py with proxy heuristics) or AI judge (LLM rubric scoring). Uses four-way separation in both modes. Inspired by Karpathy's autoresearch.

AutoResearch — Autonomous Optimization Loop

USE WHEN the user runs `/autoresearch`, says "autoresearch", "optimize this prompt", "improve this overnight", "run an optimization loop", "iterate on this", "auto-improve", or wants to autonomously refine a file against measurable criteria.

---

Architecture: Separation of Roles

This skill supports two mutually exclusive evaluation modes. Both maintain four-way separation.

Deterministic Mode (default) — Four-Way Separation

| Role | Who | Knows Eval Code? | Knows Prompt History? | |------|-----|-----------------|----------------------| | **Main Agent** | You (optimizer) | **NO** — reads metric number only | Yes — reads logs, plans changes | | **Eval Agent** | `autoresearch-eval-agent` sub-agent | Yes — writes eval.py | No | | **Test Runner** | `autoresearch-test-runner` sub-agent | **NO** — fresh context | **NO** | | **Judge Script** | `eval.py` (deterministic Python) | IS the eval | No |

Metric: `pass_rate`. Best for tasks with clear, mechanical quality checks (word count, format, keywords, structure).

AI Judge Mode (opt-in) — Four-Way Separation

| Role | Who | Knows Rubric? | Knows Prompt History? | |------|-----|---------------|----------------------| | **Main Agent** | You (optimizer) | Criteria names only | Yes — reads logs, plans changes | | **Eval Agent** | `autoresearch-eval-agent` sub-agent | Yes — writes rubric.md | No | | **Test Runner** | `autoresearch-test-runner` sub-agent | **NO** — fresh context | **NO** | | **Judge Agent** | `autoresearch-judge` sub-agent | **Follows rubric** | **NO** — fresh context |

Metric: `quality_score`. Best for creative/subjective tasks where mechanical checks miss the point (tone, authenticity, narrative, emotional resonance).

**The modes are mutually exclusive.** Deterministic mode uses eval.py only. AI judge mode uses the judge agent only. There is no combined score.

Key isolation (both modes):

• The **optimizer** never writes eval artifacts, never generates outputs
• The **eval agent** writes the eval system once (eval.py OR rubric.md), then disappears
• The **test runner** never sees the eval or rubric — fresh context every time
• The **judge** (script or agent) evaluates without knowing iteration history or optimizer intent

---

User Interaction: Always Use AskUserQuestion

**Whenever you need user input or confirmation, use the `AskUserQuestion` tool.** This shows an interactive popup in Cowork.

**CRITICAL: Keep popup text SHORT.** The popup is small — long text becomes unreadable. Follow this pattern:

1. **Write details in chat FIRST** (assertions list, eval code, explanations) 2. **Then use AskUserQuestion with ONLY a short question** — one sentence max

**Good:**

[In chat]: Here are 7 proposed assertions:
1. Covers all 4 search clusters
2. Each entry has title, channel, views
3. Contains opportunity mapping
...

[AskUserQuestion]: "Do these assertions look right?"
Options: "These look good" / "Adjust some" / [free text]

**Bad:**

[AskUserQuestion]: "Here are 7 proposed assertions for what a 'good' output looks like: 1. Covers all 4 search clusters (Core, Tools, Niche, Competitors)... [giant paragraph]"

Never put lists, explanations, or details inside AskUserQuestion. The popup is for the QUESTION only.

---

The Three Rules

Every criterion — whether proposed by you or provided by the user — MUST pass these three rules before entering the loop. This is the most important step in the process. Bad criteria produce bad evals, and bad evals waste every iteration.

**Rule 1: State the exact condition, not the goal.** Don't describe what you want. Describe what you can measure.

| Bad | Good | |-----|------| | "Make sure the hook is short" | "The first line must be under 136 characters including spaces" | | "Should be professional" | "Contains no exclamation marks and no ALL CAPS words (3+ letters)" | | "Include relevant data" | "Contains at least one specific number or statistic with a source" |

**Rule 2: One criterion, one variable.** Each criterion tests exactly one thing. If you're tempted to use "and" to connect two checks, split them into two separate criteria.

| Bad | Good | |-----|------| | "Under 150 words and ends with a question" | Criterion 1: "Under 150 words" / Criterion 2: "Last sentence ends with a question mark" | | "Professional tone with no jargon" | Criterion 1: "No words from the banned jargon list" / Criterion 2: "No sentences over 25 words" |

**Rule 3: Define the test (optional).** Describe how to verify the criterion — what to count, what regex to match, what structure to look for. This helps the eval agent write better checks and helps the judge agent score more consistently.

| Criterion | Test definition | |-----------|----------------| | "First line under 136 characters" | `len(lines[0]) <= 136` | | "Contains at least one statistic" | `re.search(r'\d+[%x]?\s', text)` returns a match | | "Ends with a question" | `text.rstrip().endswith("?")` |

If the user provides criteria that violate The Three Rules, rewrite them — show the user the before/after so they understand the improvement.

---

When the User Pastes Content

1. Save it to a file in the working directory (e.g., `target-skill.md`)

2. **Explain The Three Rules to the user first.** Before proposing any criteria, write in chat:

> **The Three Rules** — every criterion must pass these before we start: > > 1. **State the exact condition, not the goal.** "First line under 136 characters" not "keep the hook short." > 2. **One criterion, one variable.** If it has "and", split it into two. > 3. **Define the test (optional).** How to