/optimization-mission-playbook

name: optimization-mission-playbook
description: "Domain playbook for optimization missions — any task whose goal is to move a metric: performance, latency, throughput, memory, cost, score, quality, compression, ranking, solver, model/eval, and similar metric-improvement work. Defines how to think about and run an optimization mission: establishing the ground truth, trusting the measurement, profiling to the dominant cost, estimating the ceiling, generating and pruning disposable hypotheses, exploring cheaply before verifying expensively, guarding against metric gaming and overfitting, reasoning about correctness and trade-offs, and stopping on evidence."

Optimization Mission Playbook

What This Playbook Is For

Load this playbook when the mission's goal is to **move a metric**: make something faster, smaller, cheaper, higher-scoring, more accurate, higher-throughput, better-compressed, better-ranked, or closer to a solver/objective bound. The signal is a number with a direction and a workload that produces it.

Do not use it for missions whose goal is durable behavior (a feature, a port, a migration, an API) — those are engineering missions; use `engineering-mission-playbook`. When a mission has both — build a benchmark harness *then* optimize a hot path, port a library *then* tune it — load both playbooks and record the boundary. Durable behavior and scaffolding may use engineering assertions; metric search should normally run as targetless optimization work, with formal assertions reserved for genuine durable commitments or externally required sign-off.

This playbook has two halves. **The strategy** (Posture through Anti-patterns) is how to think about an optimization mission — read it first; it is the reasoning every decision below depends on. **The method** (Operating Order onward) is how to express that thinking as orchestrator investigation, targetless experiment work in the current checkout, ledger review, and patches in this runtime. The method references the runtime lifecycle and task schema but does not redefine them.

Posture

An optimization mission moves a metric. You are not building toward a known end state — you are running an experiment whose answer you do not yet know, against a measurement that can mislead you. Think like a skeptical experimentalist, not a builder: most of the work is deciding *what to measure, where, and whether the number is real* — not deciding how to change the code.

Hold two ideas at once:

• The metric is the goal.
• The metric is a proxy that can lie — through noise, through non-comparable measurement, through gaming, through overfitting the cases you happened to look at.

A real win moves the metric **and** survives that skepticism: correctness preserved, guardrails intact, comparison fair, improvement reproducible, and the gain generalizes beyond the cases it was tuned on. A number that goes up by breaking any of those is not a win.

Honest "no win" is a valid outcome when the search was credible, bounded, and recorded. Do not manufacture a win, and do not let effort spent become evidence of progress.

Three Things You Must Never Conflate

Most optimization confusion comes from collapsing three separate concepts. Name them apart and keep them apart for the whole mission:

• **The working point** — where your current best candidate sits; the parent you launch the next experiment from. It moves every time you promote a win.
• **The scoring reference** — the fixed yardstick the result is judged against. It does not move; if it moves, every past and future comparison is corrupted.
• **The correctness oracle** — the strongest available source of truth for "still correct." It is almost never "whatever the current code happens to output." It is an explicit spec, a reference implementation, a known-good output, a differential run, or a documented verifier — chosen for independence from the candidate.

When someone says "compare against the baseline," ask *which* of these three they mean. A measurement that uses the working point as its scoring reference, or the candidate's own output as its correctness oracle, proves nothing.

Establish The Ground Truth First

Before the first experiment, map the terrain deeply. You cannot search a space you have not measured, and in optimization the two easiest ways to fool yourself are weak correctness and weak measurement. Deep investigation must produce two reusable protocols with evidence, not assumption:

• **Correctness protocol** — the strongest available oracle, how independent it

is from the candidate, the case set, adversarial/edge/stress coverage, tolerances, invariants, missing-output behavior, per-case failure handling, candidate-specific evidence requirements, and protected oracle, verifier, fixture, golden-output, and reference paths. If the candidate can pass by matching its own output, a cached public output, a happy-path sample, or an aggregate score while breaking individual cases, the correctness protocol is not ready.

• **Metric protocol** — metric name, direction (higher or lower wins), unit,

exact command/procedure, workload/case mix, scoring reference, working point, candidate binding, build/cache/input state, repetitions, aggregation, warmup/cold-state handling, seeds, timeout, resource limits, per-case vs aggregate reporting, noise band/MDE, and protected benchmark, scorer, fixture, seed, judged-set, and data paths. If you cannot reproduce the number by hand and compare baseline vs candidate apples-to-apples, the metric protocol is not ready.

Also establish:

• **The workload** — what inputs the metric and correctness protocols cover, and

whether they represent what is actually used or scored. Optimizing a toy workload that does not match the scored one is wasted from the start.

• **The scoring reference and the correctness oracle** — identified separately,

per the three-way distinction above. For the oracle, rank the candidate sou