/benchmark-sandbox

name: benchmark-sandbox
description: Run vercel-plugin eval scenarios in Vercel Sandboxes instead of local WezTerm panels. Provisions ephemeral microVMs with Claude Code + plugin pre-installed, runs benchmark prompts, extracts hook artifacts, and produces coverage reports.

Benchmark Sandbox — Remote Eval via Vercel Sandboxes

Run benchmark scenarios inside Vercel Sandboxes — ephemeral Firecracker microVMs with node24. Each sandbox gets a fresh Claude Code + Vercel CLI + agent-browser install, the local vercel-plugin uploaded, and runs a **3-phase eval pipeline**:

• **Phase 1 (BUILD)**: Claude Code builds the app with `--dangerously-skip-permissions --debug`
• **Phase 2 (VERIFY)**: A follow-up Claude Code session uses `agent-browser` to walk through user stories, fixing issues until all pass (20 min timeout)
• **Phase 3 (DEPLOY)**: A third Claude Code session links to vercel-labs, runs `vercel deploy`, and fixes build errors (up to 3 retries). Deployed apps have deployment protection enabled by default.

Skills are tracked across **all 3 phases** — each phase may trigger additional skill injections as new files/patterns are created. After each phase, a **haiku structured scoring step** (`claude -p --json-schema --model haiku`) evaluates the results as structured JSON.

Proven Working Script

Use `run-eval.ts` — the proven eval runner:

# Run default scenarios with full 3-phase pipeline
bun run .claude/skills/benchmark-sandbox/run-eval.ts

# With dynamic scenarios from a JSON file (recommended — see "Dynamic Scenarios" below)
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios-file /tmp/my-scenarios.json

# Keep sandboxes alive overnight with public URLs
bun run .claude/skills/benchmark-sandbox/run-eval.ts --keep-alive --keep-hours 8

# Build-only (skip verification and deploy)
bun run .claude/skills/benchmark-sandbox/run-eval.ts --skip-verify --skip-deploy

# Run specific scenarios by slug
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios splitwise-clone,calendly-clone

CLI Flags

| Flag | Default | Description | |------|---------|-------------| | `--concurrency N` | 5 | Max parallel sandboxes (max 10) | | `--timeout MS` | 1800000 (30 min) | Per-phase timeout in ms | | `--keep-alive` | off | Keep sandboxes running after eval | | `--keep-hours N` | 8 | Hours to keep alive (with `--keep-alive`) | | `--skip-verify` | off | Skip the agent-browser verification phase | | `--skip-deploy` | off | Skip the Vercel deploy phase | | `--scenarios a,b,c` | all | Only run specific scenarios by slug | | `--scenarios-file path` | — | Load scenarios from a JSON file instead of built-in defaults |

Dynamic Scenarios (Recommended Approach)

Instead of hardcoding tech-specific prompts, generate scenarios dynamically as a JSON file. Prompts should describe **real-world apps people want to build** using user stories — no tech name-dropping. Let the plugin figure out what Vercel tech to inject.

Scenario JSON Format

[
  {
    "slug": "pet-adoption-board",
    "prompt": "Build me a pet adoption listing board where shelters can post animals...",
    "expectedSkills": ["ai-sdk", "nextjs", "shadcn", "vercel-functions"],
    "userStories": [
      "As a visitor, I can see a grid of pet listings with photos and names",
      "As a visitor, I can click a pet card to see a detail page",
      "As a visitor, I can filter pets by type"
    ]
  }
]

Each scenario needs: `slug` (string), `prompt` (string), `expectedSkills` (string[]), `userStories` (tuple of exactly 3 strings).

Prompt Design Guidelines

• Focus on **what the user wants**, not what tech to use
• Describe real-world apps that solve real problems with friendly, stylish UX
• Include AI features naturally (recommendations, analysis, generation)
• Always end with: `"Link the project to my vercel-labs team. After building all files, start the dev server on port 3000 with \`npx next dev --port 3000\`."`
• Include storage needs (photos, uploads) to trigger vercel-storage
• Include scheduled tasks (reminders, cleanup) to trigger cron-jobs
• Include auth/middleware to trigger routing-middleware

Structured Scoring (Haiku)

Each phase gets a structured JSON score via `claude -p --json-schema --model haiku --setting-sources ""` running inside the sandbox. This is a separate quick pass — no tools, no hooks — just reads the phase output and returns structured data.

Build Score Schema

{
  "completeness": "complete|partial|minimal|empty",
  "hasApiRoutes": true,
  "hasUIComponents": true,
  "hasAIFeature": true,
  "devServerRunning": true,
  "missingFeatures": ["feature1"],
  "summary": "Brief assessment"
}

Verify Score Schema (per user story)

{
  "stories": [
    { "index": 1, "status": "pass|fail", "reason": "Evidence from output" }
  ]
}

Deploy Score Schema

{
  "deployed": true,
  "url": "https://xxx.vercel.app",
  "buildSucceeded": true,
  "errors": [],
  "summary": "Brief assessment"
}

**Important**: The `claude -p --output-format json` response wraps results — the actual schema data is in `parsed.structured_output`, not the top-level object.

Critical Sandbox Environment Facts

| Property | Value | |----------|-------| | Home directory | `/home/vercel-sandbox` (NOT `/home/user/` or `/root/`) | | User | `vercel-sandbox` (NOT `root`) | | Claude binary | `/home/vercel-sandbox/.global/npm/bin/claude` | | PATH (via sh -c) | Includes `~/.global/npm/bin` — claude findable by name | | Port exposure | `sandbox.domain(3000)` → `https://subdomain.vercel.run` | | Snapshot persistence | **Files AND npm globals survive** snapshot restore — use `sandbox.snapshot()` → `Sandbox.create({ source: { type: "snapshot", snapshotId } })` | | SDK version | `@vercel/sandbox@1.8.0` (v2 beta's named sandbox endpoint returns 404 for this team) | | Team tier | Enterprise (vercel-labs) — **no known sandbox time cap** |

Key Discoveries (Hard-Won)

1. *