/spec-optimize

name: spec-optimize
description: "Run metric-driven iterative optimization loops. Define a measurable goal, build measurement scaffolding, then run parallel experiments that try many approaches, measure each against hard gates and/or LLM-as-judge quality scores, keep improvements, and converge toward the best solution. Use when optimizing clustering quality, search relevance, build performance, prompt quality, or any measurable outcome that benefits from systematic experimentation. Inspired by Karpathy's autoresearch, generalized for multi-file code changes and non-ML domains."
argument-hint: "[path to optimization spec YAML, or describe the optimization goal]"

Iterative Optimization Loop

Run metric-driven iterative optimization. Define a goal, build measurement scaffolding, then run parallel experiments that converge toward the best solution.

Workflow Contract Summary

When To Use

Use when a measurable outcome can improve through iterative experiments, hard gates, and/or LLM-as-judge scoring.

When Not To Use

Do not use for ordinary implementation, vague improvement requests without a metric, debugging without a feedback loop, or unbounded spend/concurrency. Name the destination when routing out: a bug with no stable repro and no measurement loop is `spec-debug`'s work (or `spec-work` for settled implementation) — diagnosing the bug inside this workflow to turn it into an optimization goal is adopting the wrong workflow, not adapting it; route out explicitly instead of drafting a spec around the diagnosis.

Inputs

An optimization spec or goal, mutable/immutable scope, measurement command or scaffold plan, budget limits, experiment settings, repository instructions, and baseline evidence.

Outputs

A measurement scaffold and experiment log, scored experiment results, kept/rejected variants, final integrated changes when appropriate, and post-run recommendations.

Artifacts

Run state under `.spec-first/workflows/spec-optimize/<spec-name>/`, experiment worktrees/results, strategy digests, and no hidden workflow state outside the documented log.

Failure Modes

Missing metric, missing measurement command, unsafe scope, excessive or uncapped budget, failed baseline, write verification failure, or unavailable dispatch/worktree backend.

Workflow

Validate the spec and budget, establish the baseline, run bounded experiments, measure and write results immediately, select winners, integrate only verified improvements, and summarize evidence.

Downstream Consumers

Code review、benchmark maintainer、在性能/相关性变更时参与的 release reviewer，以及检查 experiment logs 的人工审查者。

Scenario Capability

Follows `docs/contracts/workflows/scenario-capability-matrix.md` (default). Overrides: none

Interaction Method

Use the platform's blocking question tool: `AskUserQuestion` in Claude Code (call `ToolSearch` with `select:AskUserQuestion` first if its schema isn't loaded) or `request_user_input` in Codex. Fall back to numbered options in chat only when no blocking tool exists in the harness or the call errors (e.g., Codex edit modes) — not because a schema load is required. Never silently skip the question.

Input

<optimization_input> #<invocation arguments supplied by the current host> </optimization_input>

If the input above is empty, ask: "What would you like to optimize? Describe the goal, or provide a path to an optimization spec YAML file."

Optimization Spec Schema

Reference the spec schema for validation:

`references/optimize-spec-schema.yaml`

Experiment Log Schema

Reference the experiment log schema for state management:

`references/experiment-log-schema.yaml`

Quick Start

For a first run, optimize for signal and safety, not maximum throughput:

• Start from `references/example-hard-spec.yaml` when the metric is objective and cheap to measure
• Use `references/example-judge-spec.yaml` only when actual quality requires semantic judgment
• Prefer `execution.mode: serial` and `execution.max_concurrent: 1`
• Cap the first run with `stopping.max_iterations: 4` and `stopping.max_hours: 1`
• Avoid new dependencies until the baseline and measurement harness are trusted
• For judge mode, start with `sample_size: 10`, `batch_size: 5`, and `max_total_cost_usd: 5`

For a friendly overview of what this skill is for, when to use hard metrics vs LLM-as-judge, and example kickoff prompts, see:

`references/usage-guide.md`

Admission And Budget Gate

Do not run `spec-optimize` as an expensive substitute for ordinary work. Before Phase 1, confirm the run has all of these:

• A repeatable measurement target: `metric.primary.type`, `metric.primary.name`, and `metric.primary.direction`
• At least one cheap degenerate gate that rejects broken variants before judging or ranking
• A measurement command, or a concrete plan to build the harness before any experiment is dispatched
• Explicit `scope.mutable` and `scope.immutable` boundaries
• Explicit experiment budget: `stopping.max_iterations`, `stopping.max_hours`, and `stopping.plateau_iterations`
• Execution budget: `execution.mode` and `execution.max_concurrent`
• For judge mode, a finite `metric.judge.max_total_cost_usd`, unless the user explicitly approves uncapped spend

If any item is missing, stop and help the user create a safe spec or route the work to the current host's plan/work/debug entrypoint instead. Do not continue with an open-ended optimization loop.

Measurement-Only Calibration Mode

When the invocation or approved spec selects `mode:measurement-only`, load `references/measurement-only-calibration.md`. This mode compares an explicitly identified baseline and candidate against the same repeatable task/corpus. It must run an A/A noise floor before A/B, use a pre-registered acceptance threshold, classify broken runs separately from regressions, and produce only a measurement artifact plus stop/defer guidance.

Before the first measurement, materialize the approved frozen inputs as a run-lo