/run-mcp-evals

description: Run golden-test eval suites against one or all MCP servers in mcp-servers/; compares actual tool responses to expected via exact-match + regex + min-count tiers; supports mock-mode so CI runs without API keys
argument-hint: [<server-name>] [--ci] [--mock-only] [--live-only]
allowed-tools: Bash, Read

Run MCP server eval suites. Spawns each server via stdio, exercises tool calls, and scores responses against `mcp-servers/<name>-mcp/evals/*.json` golden tests. `--ci` exits nonzero on any failure. `--mock-only` skips tests without a `mock_response`; `--live-only` ignores `mock_response` and hits the real API.

**Purpose** — catch regressions in generated MCP servers before they hit production; enforce golden tests as the contract between forge output and real API behaviour; run safely in CI without API keys via mock-mode.

**When to use** — after `/forge-from-openapi` or `/migrate-to-hardened`; on CI push (always `--mock-only`); when an MCP returns unexpected errors; before publishing a new MCP version (use `--live-only` or default hybrid).

**Inputs**

• `<server-name>` — optional; run only this server (e.g. `resend`, `stripe`). Omit to scan all.
• `--ci` — machine-readable NDJSON to stdout; human summary to stderr; exit 1 on any failure.
• `--mock-only` — only run tests that have a `mock_response` field; skip live tests. Use in CI to avoid requiring API keys.
• `--live-only` — ignore `mock_response` on all tests; always call the live API. Use for manual pre-release verification.
• Default (neither flag): prefer mock when `mock_response` is present; call live API when not present (hybrid mode).

**Mock-mode overview** — See `rules/eval-mock-mode-discipline.md` for the full discipline. Short version: the harness sets `MCP_MOCK_RESPONSE_JSON=<base64>` in the MCP server's env before spawning it; the server's fetch wrapper checks this var and returns the canned response without touching the network. This requires the MCP server to implement the env-var hook (added automatically by `forge-from-openapi --harden`).

---

Step 1 — Discover servers + parse flags

PLUGIN_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd 2>/dev/null)" \
  || PLUGIN_ROOT="${HOME}/.claude/plugins/heymegabyte-claude-skills"

SERVER_NAME=""
CI_MODE=0
MOCK_ONLY=0
LIVE_ONLY=0

for arg in "$@"; do
  case "$arg" in
    --ci)        CI_MODE=1 ;;
    --mock-only) MOCK_ONLY=1 ;;
    --live-only) LIVE_ONLY=1 ;;
    --*)         echo "Unknown flag: $arg" >&2; exit 2 ;;
    *)           SERVER_NAME="$arg" ;;
  esac
done

if [[ $MOCK_ONLY -eq 1 && $LIVE_ONLY -eq 1 ]]; then
  echo "✗ --mock-only and --live-only are mutually exclusive" >&2
  exit 2
fi

# Glob all MCP server dirs
mapfile -t MCP_DIRS < <(ls -d "${PLUGIN_ROOT}/mcp-servers/"*-mcp 2>/dev/null | sort)

if [[ ${#MCP_DIRS[@]} -eq 0 ]]; then
  echo "✗ No MCP server dirs found under ${PLUGIN_ROOT}/mcp-servers/" >&2
  exit 1
fi

# Filter to named server if provided
if [[ -n "$SERVER_NAME" ]]; then
  mapfile -t MCP_DIRS < <(printf '%s\n' "${MCP_DIRS[@]}" | grep "/${SERVER_NAME}-mcp$" || true)
  if [[ ${#MCP_DIRS[@]} -eq 0 ]]; then
    echo "✗ No server found matching '${SERVER_NAME}'" >&2
    exit 1
  fi
fi

---

Step 2 — For each server: locate entry point and eval files

For each `MCP_DIR` in `MCP_DIRS`:

1. Derive `SERVER_ID` = basename of `MCP_DIR` (e.g. `resend-mcp`). 2. Locate entry point in priority order:

• `${MCP_DIR}/mcp-server/dist/index.js` — prefer built artifact
• `${MCP_DIR}/mcp-server/src/index.ts` — fallback; requires `tsx` or `ts-node`
• Skip server with warning if neither exists.

3. Glob eval files: `${MCP_DIR}/evals/*.json`. If none exist, emit `SKIP (no evals)` and continue.

for MCP_DIR in "${MCP_DIRS[@]}"; do
  SERVER_ID=$(basename "$MCP_DIR")
  ENTRY=""
  if [[ -f "${MCP_DIR}/mcp-server/dist/index.js" ]]; then
    ENTRY="${MCP_DIR}/mcp-server/dist/index.js"
    RUNNER="node"
  elif [[ -f "${MCP_DIR}/mcp-server/src/index.ts" ]]; then
    ENTRY="${MCP_DIR}/mcp-server/src/index.ts"
    RUNNER="npx tsx"
  else
    echo "  SKIP ${SERVER_ID} — no dist/index.js or src/index.ts" >&2
    continue
  fi

  mapfile -t EVAL_FILES < <(ls "${MCP_DIR}/evals/"*.json 2>/dev/null | sort)
  if [[ ${#EVAL_FILES[@]} -eq 0 ]]; then
    echo "  SKIP ${SERVER_ID} — no evals/*.json" >&2
    continue
  fi
done

---

Step 3 — Determine mock vs. live per test, then spawn + run

For each test object read from the eval JSON:

3a — Mock/live resolution

USE_MOCK = false

if LIVE_ONLY:
  USE_MOCK = false                          # always hit live API
elif test.mock_response is present:
  if MOCK_ONLY or default:
    USE_MOCK = true                         # prefer mock when available
elif MOCK_ONLY and test.mock_response is absent:
  SKIP test with note "no mock_response, --mock-only set"
  continue
# else: live-only (no mock_response) in default mode → USE_MOCK = false

Emit a per-test mode badge in human output:

• `[mock]` — running against mock_response
• `[live]` — calling the real API
• `[skip/no-mock]` — skipped because --mock-only but no mock_response

3b — Spawn the MCP server

When `USE_MOCK = true`, base64-encode the `mock_response` object and set it in the server env:

MOCK_JSON=$(python3 -c "import json,base64,sys; d=json.load(sys.stdin); print(base64.b64encode(json.dumps(d).encode()).decode())" <<< '{"status":200,"body":{...}}')
export MCP_MOCK_RESPONSE_JSON="$MOCK_JSON"

Then spawn the server as normal:

$RUNNER "$ENTRY" &
SERVER_PID=$!
sleep 2  # give server time to boot

When `USE_MOCK = false`, spawn without the env var (or explicitly unset it):

unset MCP_MOCK_RESPONSE_JSON
$RUNNER "$ENTRY" &
SERVER_PID=$!
sleep 2

**Option A (recommended, requires server cooperation):** The MCP server's fetch wrapper checks `process.env.MCP_MOCK_RESPONSE_JSON` at the start of every outgoing HTTP call. If present, it decodes and returns the canned response without touc