bakeoff

bakeoff

Turn one hard "which should I do?" into a judged tournament — and get back a defensible winner.

Terminal summary from a real run — verdict and scores are from the saved report, re-rendered at a readable pace.

bakeoff is a Claude Code skill. Hand it a decision and it generates diverse candidate solutions, auto-derives the criteria that matter for that specific problem (so you don't have to know what to score on), judges every candidate with independent scorers, and returns the winner plus a ranked shortlist — with the reason each one won or lost. You just type your request; it infers the shape (a comparison, an improvement, an idea, a "what if we do X", a problem, or a scoping call), the depth, and the rubric — no flags.

The hard part of any comparison isn't the scoring — it's knowing what to evaluate. bakeoff derives the rubric for you. Here's the run shown in the GIF above:

/bakeoff "8h/24h email deadline: Batches vs keep-sync vs merge?"

Roles  → status-quo · cost-first · dedup · max-savings
Rubric → Deadline-fit 28 · COGS 24 · Personalization 16 ·
         Complexity 16 · Ops 12 · Reversibility 4  [approve? yes]
Judges ×2 → reconcile → shortlist
   B  keep sync per-cohort   73   wins deadline-fit; 0 on COGS
   A  Message Batches        70   50% off, tiny in absolute $
   C  embedding merge        69
   D  hybrid                 55
Refute → both judges picked A, but the 50% cut is half of an
         already-tiny bill (~$30–150/mo) → flips to B
Winner → B: keep sync now; escalate 24h projects past ~$200/mo

You never supplied the six dimensions or their weights — and the adversarial pass caught that the judges' pick rested on a saving too small to matter. That's the point.

Why it's built this way

• Select, then graft. Diverse candidates + judge-based selection is the core. But when the top two are strong on different dimensions, it grafts the runner-up's best element into the winner — kept only if it re-scores above the best single candidate. In practice the graft wins or sharpens the pick about as often as it's discarded.
• Diversity is the biggest lever. Each candidate generator gets a distinct, problem-specific role (e.g. cost-first vs status-quo), so the field genuinely spans the space.
• Independent judges, mechanically reconciled — not a debate (debate amplifies bias). Two judges score independently; a deterministic script merges them with a lower-score rule on disagreements.
• Position-bias controlled. Each judge sees the candidates in a different shuffled order, referenced by stable IDs.
• The leader gets stress-tested. Before committing, an adversarial pass actively tries to refute the top candidate. A plausible-but-wrong winner shouldn't survive.
• Grounded when it matters. For decisions that hinge on real facts (a library version, your actual codebase), it reads/searches before judging — and flags any part it couldn't verify as "training-knowledge only."

Install

Pick one — all three install the same self-contained skill.

Agent Skills CLI (works with Claude Code, Codex, Cursor):

npx skills add CoriChui/bakeoff

Claude Code plugin (marketplace install, auto-updates):

/plugin marketplace add CoriChui/bakeoff
/plugin install bakeoff@bakeoff

Manual — clone straight into your skills directory:

git clone https://github.com/CoriChui/bakeoff.git ~/.claude/skills/bakeoff

Then in Claude Code:

/bakeoff "which caching strategy should we use for the API layer?"

That's it. The skill is self-contained — the rubric-builder, scorer, and reconciliation script are bundled (adapted from the evaluate skill). No other skills required.

Requirements

• Claude Code (skills support).
• Node.js on your PATH — the score reconciler (scripts/reconcile-scores.js) runs under Node.

When to use it

Reach for bakeoff when all three hold:

1. Wide solution space — several genuinely defensible approaches, not one obvious answer.
2. Costly to reverse — a wrong call is expensive to unwind.
3. Unclear criteria — you can't easily say why one option should beat another.

Good fits: architecture choices, library/database/tool selection, refactor strategies, migration approaches, "is the AI's suggestion actually good, or is there something better?" — and you needn't phrase any of them as a "decision." Any request shape qualifies: a comparison ("X vs Y"), an improvement ("best way to X"), an idea ("what should we build"), a proposal ("what if we do X"), a problem ("how do we handle this"), or a scoping call ("minimal X before launch"). bakeoff recognizes the shape itself.

Don't use it when a test, type-check, or lint settles the question, or when you're scoring a single artifact with no alternatives — that's a job for a plain evaluation, not a tournament. (Low-stakes or single-obvious-answer "which/how" questions are a direct answer, not a tournament — the three-part gate above still governs.)

How it decides — no flags needed

You just type your request. bakeoff infers the rest:

• Entry point — it almost always generates the candidates itself. If your prompt names a concrete option or plan ("should the hero be GitHub-only?", "A vs B vs C"), that becomes a candidate and it invents rivals around it. Either way it always adds rivals — you never just judge what you brought.
• Floor candidate — for a proposal / idea / scoping request it always includes a genuine defer / do-the-minimal / status-quo candidate, and the rubric rewards restraint, so "not yet" can win (in real runs it often does).
• Depth — auto-scaled to the grounded stakes of the call, not the surface of the question:

Power users can still override — --lean / --thorough (depth), --seed <plan> (force a plan as Candidate A), --compare (judge pasted candidates) — but none are required. See SKILL.md for the full pipeline.

How it works

FRAME → { GENERATE K candidates  ∥  BUILD RUBRIC } → [rubric gate] →
   JUDGE (2 scorers ∥, randomized order, reconciled) →
   RANK → ADVERSARIAL CHECK (only if the top-two are close or the leader is suspect) →
   winner + top-N shortlist (+ optional synthesis) → REPORT

The rubric is built in parallel with candidate generation and blind to the candidates — so it describes the decision, not whichever option it might otherwise favor. Every run ends with a saved report (docs/bakeoffs/YYYY-MM-DD-<slug>.md) containing the recommendation, the shortlist, the full score matrix, judge agreement, and the rubric — so the decision is auditable and reusable later.

See full saved reports — a real run and an illustrative one — in examples/.

License

MIT