/benchmark-agents

name: benchmark-agents
description: Advanced AI agent benchmark scenarios that push Vercel's cutting-edge platform features — Workflow SDK, AI Gateway, MCP, Chat SDK, Queues, Flags, Sandbox, and multi-agent orchestration. Designed to stress-test skill injection for complex, multi-system builds.

Benchmark Agents — Advanced AI Systems

Launch real Claude Code sessions with the plugin installed, verify skill injection, monitor PostToolUse validation catches, and produce a coverage report. This skill covers the full eval loop: setup → launch → monitor → verify → fix → release → repeat.

How Evals Work (The Only Correct Method)

Evals are run by **you, in this conversation**, not by scripts. The process is:

1. You create directories and install the plugin via Bash tool calls 2. You spawn WezTerm panes with `wezterm cli spawn` — each pane runs an independent Claude Code interactive session 3. You wait, then check debug logs and claim dirs to see what the plugin injected 4. You inspect the generated source code for correctness 5. You read conversation logs to find what the user had to correct 6. You update skills/hooks, run `/release`, and spawn more evals

**Never use `claude --print`, eval scripts, or `Bun.spawn(["claude", ...])`**. These do not work because:

• Plugin hooks (PreToolUse, PostToolUse, UserPromptSubmit) only fire during interactive tool-calling sessions
• `--print` mode generates text without executing tools — no files are created, no deps installed, no dev servers started
• No `session_id` means dedup, profiler, and claim files don't work

**The WezTerm interactive approach is the only method that exercises the plugin correctly.** Every eval in our history (60+ sessions) used this approach.

DO NOT (Hard Rules)

These are **absolute prohibitions**. Violating any of them wastes the entire eval run:

• **DO NOT** use `claude --print` or `-p` flag — hooks don't fire, no files created
• **DO NOT** use `--dangerously-skip-permissions` — changes agent behavior
• **DO NOT** create projects in `/tmp/` — always use `~/dev/vercel-plugin-testing/`
• **DO NOT** manually create `settings.local.json` or wire hooks by hand — use `npx add-plugin`
• **DO NOT** set `CLAUDE_PLUGIN_ROOT` manually — the plugin manages this
• **DO NOT** use `bash -c` or `bash -lc` in WezTerm — always use `/bin/zsh -ic`
• **DO NOT** use the full path to claude — use the `x` alias (it's configured in zsh)
• **DO NOT** create custom `debug.log` files with stderr redirects — debug logs go to `~/.claude/debug/`
• **DO NOT** write eval runner scripts in TypeScript/JavaScript — do everything as Bash tool calls in the conversation
• **DO NOT** try to `git init` or create `package.json` manually — `npx add-plugin` + the WezTerm session handle all scaffolding
• **DO NOT** use uppercase letters in directory names — npm rejects them (e.g. `T` in timestamps breaks `create-next-app`)

**Copy the exact commands below. Do not improvise.**

Setup & Launch (Exact Commands)

Naming convention

**Always append a timestamp** to directory names so reruns don't overwrite old projects:

<slug>-<yyyymmdd>-<hhmm>

Example: `tarot-card-deck-20260309-1227`, `interior-designer-20260309-1227`

Generate the timestamp with: `date +%Y%m%d-%H%M`

1. Create test directory and install plugin

TS=$(date +%Y%m%d-%H%M)
SLUG="my-app-$TS"
mkdir -p ~/dev/vercel-plugin-testing/$SLUG
cd ~/dev/vercel-plugin-testing/$SLUG
npx add-plugin https://github.com/vercel/vercel-plugin -s project -y

2. Launch session via WezTerm

wezterm cli spawn --cwd /Users/johnlindquist/dev/vercel-plugin-testing/$SLUG -- /bin/zsh -ic \
  "unset CLAUDECODE; VERCEL_PLUGIN_LOG_LEVEL=debug x '<PROMPT>' --settings .claude/settings.json; exec zsh"

Key flags:

• `unset CLAUDECODE` — prevents nested session detection error
• `VERCEL_PLUGIN_LOG_LEVEL=debug` — enables hook debug output in `~/.claude/debug/`
• `x` — alias for `claude` CLI
• `--settings .claude/settings.json` — loads project-level plugin settings

3. Find the debug log (wait ~25s for SessionStart hooks)

find ~/.claude/debug -name "*.txt" -mmin -2 -exec grep -l "$SLUG" {} +

4. Launch multiple sessions in parallel

Create dirs and install plugin in a loop, then spawn each WezTerm pane:

TS=$(date +%Y%m%d-%H%M)
cd ~/dev/vercel-plugin-testing
for name in tarot-deck interior-designer superhero-origin; do
  d="${name}-${TS}"
  mkdir -p "$d" && (cd "$d" && npx add-plugin https://github.com/vercel/vercel-plugin -s project -y)
done

# Then spawn each (these run in separate terminal panes)
wezterm cli spawn --cwd .../tarot-deck-$TS -- /bin/zsh -ic "unset CLAUDECODE; VERCEL_PLUGIN_LOG_LEVEL=debug x '...' --settings .claude/settings.json; exec zsh"
wezterm cli spawn --cwd .../interior-designer-$TS -- /bin/zsh -ic "unset CLAUDECODE; VERCEL_PLUGIN_LOG_LEVEL=debug x '...' --settings .claude/settings.json; exec zsh"
wezterm cli spawn --cwd .../superhero-origin-$TS -- /bin/zsh -ic "unset CLAUDECODE; VERCEL_PLUGIN_LOG_LEVEL=debug x '...' --settings .claude/settings.json; exec zsh"

Monitoring

Skill injection claims (the key metric)

TMPDIR=$(node -e "import {tmpdir} from 'os'; console.log(tmpdir())" --input-type=module)
CLAIMDIR="$TMPDIR/vercel-plugin-<session-id>-seen-skills.d"

# List all injected skills
ls "$CLAIMDIR"

# Count
ls "$CLAIMDIR" | wc -l

# Check specific skill
ls "$CLAIMDIR/workflow" && echo "YES" || echo "NO"

Hook firing

LOG=~/.claude/debug/<session-id>.txt

# SessionStart hooks
grep -c 'SessionStart.*success' "$LOG"

# PreToolUse calls and injections
grep -c 'executePreToolHooks' "$LOG"        # total calls
grep -c 'provided additionalContext' "$LOG"  # actual injections

# PostToolUse validation catches
grep 'VALIDATION' "$LOG" | head -10

# UserPromptSubmit
grep -c 'UserPromptSubmit.*success' "$LOG"

Quick status check for multiple sessions

TMPDIR=$(node -e "import {tmpdir} from