/bat-story-eval

name: bat-story-eval
description: Compare MCP tool behavior between target and baseline versions using pre-built and custom stories with diff-based triage.
disable-model-invocation: true
argument-hint: --baseline v6.6.1 [--agents gemini] [--stories s01,s02]
allowed-tools: Bash, Read, Write, Glob, Grep, Task

BAT Story Evaluation

You are the evaluator. Follow these steps IN ORDER. Do not skip steps.

Parse Arguments

From `$ARGUMENTS`, extract:

• `--baseline`: REQUIRED. Git tag/branch of the released version (e.g., `v6.6.1`).
• `--agents`: Agent list (default: `gemini`). Comma-separated.
• `--stories`: Force specific pre-built story IDs (e.g., `s01,s02`). Overrides triage selection.
• `--all-stories`: Skip triage, run ALL pre-built stories.
• `--keep-container`: Keep HA containers alive after run for manual inspection.
• `--model`: Model for Claude agent (e.g., `haiku`, `sonnet`).

If `$ARGUMENTS` is `--help` or missing `--baseline`, show usage and stop:

/bat-story-eval --baseline v6.6.1
/bat-story-eval --baseline v6.6.1 --agents gemini,claude
/bat-story-eval --baseline v6.6.1 --stories s01,s02
/bat-story-eval --baseline v6.6.1 --all-stories --agents claude --model haiku

Step 0: Triage (Diff Analysis + Custom Story Design)

0a. Compute Diff

cd "$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")/worktree/uat-stories"
git diff <baseline>..HEAD -- src/ha_mcp/ --stat
git diff <baseline>..HEAD -- src/ha_mcp/ --name-only

Classify changed files:

• **Tool modules** (`tools/tools_*.py`): specific tool implementations changed
• **Core code** (`client/`, `server.py`, `errors.py`, `tools/util_helpers.py`): affects all tools
• **Utilities** (`utils/`, `resources/`): may affect all tools
• **No src/ changes**: only tests/docs/config — select 2 smoke-test stories

0b. Select Pre-built Stories

Skip if `--stories` or `--all-stories` was passed.

1. Read the diff from 0a 2. Read all story YAMLs in `tests/uat/stories/catalog/s*.yaml` (title, description, prompt, setup) 3. For each story, reason about whether the diff could affect its outcome:

• What tools/code paths would this story exercise?
• Do any of those overlap with what changed?

4. Rules:

• Story likely exercises changed code -> **selected**
• Core code changed (`client/`, `server.py`, `errors.py`) -> **all stories selected**
• No src/ changes -> 2 representative stories as smoke test

5. Report which stories were selected and why (one sentence per story)

0c. Design Custom Stories (at least 1)

Read the diff carefully. Your job is to catch regressions. For each changed code path NOT covered by selected pre-built stories, ask: "could this break something a user would notice?" If yes, design a custom story to test that hypothesis.

**Guidelines**: Always create at least 1 custom story. Each must test a distinct regression hypothesis — don't create stories that overlap. Stop when you've covered the risky gaps.

Write each as `/tmp/custom_c<NN>.yaml` using the standard story format:

id: c01
title: "Short description of what is being tested"
category: custom
weight: 5
description: >
  Rationale: [what changed in the diff and why this scenario tests it]

setup:
  - tool: ha_config_set_helper
    args:
      helper_type: "input_boolean"
      name: "Test Entity Name"

prompt: >
  [Natural language request a real user would make that exercises the changed code]

teardown: []

verify:
  questions:
    - "Did the agent achieve the expected outcome?"
    - "Did it use the expected tools?"

expected:
  tools_should_use:
    - ha_search
  description: >
    [What a correct agent should do]

**Design principles:**

• Focus on code paths that changed in the diff
• Plausible user scenarios, not synthetic edge cases
• Setup creates realistic HA state via FastMCP in-memory steps
• Prompts are what a real user would type
• At least 1. Each tests a distinct regression hypothesis. Stop when gaps are covered.

Step 1: Run Baseline Version

For EACH agent, run all stories against the **baseline** version. One container per agent, reused across all stories.

1a. Start container with first story

cd "$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")/worktree/uat-stories"
uv run python tests/uat/stories/run_story.py \
  catalog/<first_story>.yaml \
  --agents <agent> --keep-container \
  --branch <baseline> \
  --results-file local/uat-results.jsonl

**CAPTURE from stderr**: HA URL (e.g., `http://localhost:32771`), token, session file path.

1b. Verify, then run remaining pre-built stories

After each story, verify via ha_query.py using the story's `verify.questions`:

uv run python tests/uat/stories/scripts/ha_query.py \
  --ha-url http://localhost:PORT --ha-token TOKEN \
  --agent <agent> \
  "Does an automation with alias 'Sunset Porch Light' exist?"

Record each answer as **confirmed** / **denied** / **unclear**. A non-zero exit from `ha_query.py` means the query itself failed (the output carries an `[exit N]` marker; `[exit 124]` is a timeout) — that is not one of the three outcomes; re-run it, and if it keeps failing record the story as `unverified` (Step 5) rather than scoring it. See `references/evaluation-protocol.md`.

Run remaining pre-built stories on the same container:

uv run python tests/uat/stories/run_story.py \
  catalog/<next_story>.yaml \
  --agents <agent> --ha-url http://localhost:PORT --ha-token TOKEN \
  --branch <baseline> \
  --results-file local/uat-results.jsonl

Verify each immediately after running.

1c. Run custom stories on same container

uv run python tests/uat/stories/run_story.py \
  /tmp/custom_c01.yaml \
  --agents <agent> --ha-url http://localhost:PORT --ha-token TOKEN \
  --branch <baseline> \
  --results-file local/uat-results.jsonl

Verify each via ha_query.py using the custom story's `verify.questions`.

1d. Stop container

docker stop $(docker p