/ln-45-benchmark-comparator

name: ln-45-benchmark-comparator
description: "Compares tools or implementations through controlled benchmarks and independent correctness checks."

Benchmark Comparator

**Goal:** Compare alternatives under controlled, reproducible conditions. Correctness comes before speed, and measured data must remain separate from estimates, setup cost, and interpretation.

**Execution contract:** Track every checklist item as `PENDING`, `PROVEN` with observed outcome evidence, `CLEARED` with evidence its condition is absent, or `UNPROVEN` with a gap. Reading, delegation, tool failure, zero exit status, and self-reported success are not proof. Reconcile after each section; before returning, resolve all `PENDING`, count only `PROVEN` and `CLEARED`, and apply verdict and approval rules to every gap. Preserve intent, scope, mode, and existing authorization. Complete authorized work before required external approval; ask only consequential unresolved choices. If nobody can answer, report the exact question and apply the skill's gap verdict without waiting or guessing. Cite any blocking instruction and unresolved boundary; never invent approval gates. Scale depth to material risk; preserve mandatory checks, dependencies, and safety order while choosing verification methods. Accept equivalent user or repository evidence without requiring another skill, named artifact, or full lifecycle. Preserve source requirement and decision IDs. Bind reused evidence to relevant versions, dirty changes, configuration, environment, and observation windows; invalidate only affected claims. Choose the primary workflow by outcome and explicit selection. Consume neighboring contracts and valid evidence without duplicating ownership or restarting audits. Split distinct deliverables by scope, not mandatory skill invocation. Broad independent assessments use audit workflows; focused domain reviews remain read-only. On continuation, reconcile task, authorization, state, and gaps. Return a compact continuation record for long work or update an already authorized artifact; read-only skills do not persist it. Distinguish artifact readiness, verified behavior, and external-action authority. Trace applicable requirements to their authoritative owner, planned or implemented change, acceptance evidence and status in the existing artifact or response; no new registry is required. Explicit constraints remain binding until an authorized deviation; risk determines priority, not permission to ignore them. Investigate the causal flow beyond initially named files. Within implementation authority, include structural repair needed for the outcome; minimize total complexity and risk, not changed lines. This does not authorize unrelated cleanup or mutation during review. Before retrieval, define the evidence question; read selectively without skipping applicable instructions. Use tools for mechanical processing; summarize bounded results, errors, coverage, and source paths, keeping full output outside context. Before technical decisions, establish applicability and actual versions; verify version-dependent assumptions against official documentation or specifications. Search and read primary pages for current external recommendations; snippets are not evidence. Version-matched bundled documentation or valid prior evidence suffices. Record decision, version, source, and limitations. Mark unavailable evidence UNVERIFIED; block only essential dependent work. Prefer native mechanisms, justify added tools or abstractions with evidence, and reuse the existing rule owner.

Tool Routing

| Need | Preferred tool | Use it when | Fallback | |---|---|---|---| | Canonical workload and oracle | Repository fixtures, tests, expected diffs, schemas, or independently specified outcomes | Defining what success means before either candidate runs | Create the smallest deterministic fixture that represents the decision | | Isolation | Clean Git worktrees, temporary directories, controlled environment, fixed seeds, and resettable caches | Preventing one candidate or run from contaminating another | Sequential clean-room setup with verified cleanup | | Execution | The same shell runner and wrapper for every candidate | Capturing commands, exit status, stdout, stderr, timing, and artifacts consistently | Manual execution with an explicit reproducibility limitation | | Activation proof | Logs, traces, command records, process metadata, or candidate-specific artifacts | Verifying the intended alternative actually ran and did not fall back | Treat the run as invalid when activation cannot be proven | | Correctness grading | Tests, output parser, diff, schema validation, or independent oracle | Every scenario before cost comparison | Manual blind grading against written expectations | | Performance and cost | Monotonic timer, resource metrics, token or usage telemetry, tool-call logs, and failure counts | Metrics are observable through the same method for all candidates | Label derived or estimated values and keep them out of measured aggregates | | External semantics | Official documentation and specifications | Candidate configuration or claimed behavior needs current verification | Primary-source web research; otherwise mark the claim `UNVERIFIED` |

Do not tune the scenario after observing a preferred candidate, mix measurements from different workloads, or present internal estimates as externally measured facts. Benchmarking may create temporary worktrees and artifacts but must not change the source baseline or unapproved external state.

Evidence Rules

• Hold all non-tested variables constant or record and analyze the confounder.
• Correctness failure cannot be compensated by better speed, token use, or cost unless the decision explicitly allows degraded correctness.
• Use repeated runs and report raw values, center, spread, failures, and outliers; never headline the best run.
• Keep setup or indexing cost, steady-state cost, maintenance burden, and ru