/agent-comparison

name: agent-comparison
description: "A/B test agent variants for quality and token cost."
user-invocable: false
allowed-tools:
  - Read
  - Write
  - Edit
  - Bash
  - Glob
  - Grep
  - Task
routing:
  triggers:
    - "compare agents"
    - "A/B test agents"
    - "benchmark agents"
    - "optimize skill"
    - "optimize description"
    - "run autoresearch"
  not_for: "creating new skills from scratch (use skill-creator skill) — this skill compares and optimizes existing agent descriptions and routing"
  category: meta-tooling
  pairs_with:
    - agent-evaluation
    - skill-eval

Agent Comparison Skill

Compare agent variants through controlled A/B benchmarks. Runs identical tasks on both agents, grades output quality with domain-specific checklists, and reports total session token cost to a working solution. This skill is exclusively for agent variant comparison — use `agent-evaluation` for single-agent assessment, and `skill-eval` for skill testing.

Reference Loading Table

| Signal | Load These Files | Why | |---|---|---| | selecting benchmark tasks and directory layout (Phase 1) | `benchmark-tasks.md` | Loads detailed guidance from `benchmark-tasks.md`. | | example-driven tasks, errors | `examples-and-errors.md` | Loads detailed guidance from `examples-and-errors.md`. | | scoring solutions: 5-criteria rubric and effective cost calculation | `grading-rubric.md` | Loads detailed guidance from `grading-rubric.md`. | | deciding when to run comparisons; December 2024 baseline data | `methodology.md` | Loads detailed guidance from `methodology.md`. | | configuring autoresearch: targets, task formats, eval isolation modes | `optimization-guide.md` | Loads detailed guidance from `optimization-guide.md`. | | executing Phase 5 OPTIMIZE step by step | `optimize-phase.md` | Loads detailed guidance from `optimize-phase.md`. | | writing the Phase 4 comparison report | `report-template.md` | Loads detailed guidance from `report-template.md`. |

Instructions

> See `references/examples-and-errors.md` for error handling. See `references/optimize-phase.md` for Phase 5 OPTIMIZE full procedure. See `references/methodology.md` for December 2024 benchmark data.

Phase 1: PREPARE

**Goal**: Create benchmark environment and validate both agent variants exist.

Read and follow the repository CLAUDE.md before starting any execution.

**Step 1: Analyze original agent**

wc -l agents/{original-agent}.md
grep "^## " agents/{original-agent}.md
grep -c '```' agents/{original-agent}.md

**Step 2: Create or validate compact variant**

If creating a compact variant, preserve:

• YAML frontmatter (name, description, routing)
• Core patterns and principles
• Error handling philosophy

Remove or condense:

• Lengthy code examples (keep 1-2 representative per pattern)
• Verbose explanations (condense to bullet points)
• Redundant instructions and changelogs

Target 10-15% of original size while keeping essential knowledge. Remove redundancy, not capability — stripping error handling patterns or concurrency guidance creates an unfair comparison because the compact agent is missing essential knowledge rather than expressing it concisely.

**Step 3: Validate compact variant structure**

head -20 agents/{compact-agent}.md | grep -E "^(name|description):"
echo "Original: $(wc -l < agents/{original-agent}.md) lines"
echo "Compact:  $(wc -l < agents/{compact-agent}.md) lines"

**Step 4: Create benchmark directory and prepare prompts**

mkdir -p benchmark/{task-name}/{full,compact}

Write the task prompt ONCE, then copy it for both agents. Both agents must receive the exact same task description, character-for-character, because different requirements produce different solutions and invalidate all measurements.

Keep benchmark scripts simple — no speculative features or configurable frameworks that were not requested.

**Gate**: Both agent variants exist with valid YAML frontmatter. Benchmark directories created. Identical task prompts written. Proceed only when gate passes.

Phase 2: BENCHMARK

**Goal**: Run identical tasks on both agents, capturing all metrics.

**Step 1: Run simple task benchmark (2-3 tasks)**

Use algorithmic problems with clear specifications (e.g., Advent of Code Day 1-6). Simple tasks establish a baseline — if an agent fails here, it has fundamental issues. Running multiple simple tasks is necessary because a single data point is sensitive to task selection bias and cannot distinguish luck from systematic quality.

Spawn both agents in parallel using Task tool:

Task(
  prompt="[exact task prompt]\nSave to: benchmark/{task}/full/",
  subagent_type="{full-agent}"
)

Task(
  prompt="[exact task prompt]\nSave to: benchmark/{task}/compact/",
  subagent_type="{compact-agent}"
)

Run in parallel to avoid caching effects or system load variance skewing results.

**Step 2: Run complex task benchmark (1-2 tasks)**

Use production-style problems that require concurrency, error handling, edge case anticipation — these are where quality differences emerge because simple tasks mask differences in edge case handling. See `references/benchmark-tasks.md` for standard tasks.

Recommended complex tasks:

• **Worker Pool**: Rate limiting, graceful shutdown, panic recovery
• **LRU Cache with TTL**: Generics, background goroutines, zero-value semantics
• **HTTP Service**: Middleware chains, structured errors, health checks

**Step 3: Capture metrics for each run**

Record immediately after each agent completes — delayed recording loses precision. Track input/output token counts per turn where visible, since total session cost (not just prompt size) is what matters.

| Metric | Full Agent | Compact Agent | |--------|------------|---------------| | Tests pass | X/X | X/X | | Race conditions | X | X | | Code lines (main) | X | X | | Test lines | X | X | | Session tokens | X | X | | Wall-clock time | Xm Xs | Xm Xs | | Retry cycles | X | X |

**Step 4: Run tests w