/flutter-mcp-toolkit-dogfood-iterations

name: flutter-mcp-toolkit-dogfood-iterations
description: Runs and records showcase/flutter_test_app dogfood iterations (tool_quality_rubric, run_dogfood_eval.sh, dogfood_web_eval.yaml). Use when scoring MCP/intentcall quality, appending iteration N, comparing regressions, or CI static/weekly eval gates.

<!-- @FMT_MODE_PRELUDE -->

Dogfood iterations

Tracker: `docs/evidence/dogfood/dogfood_web_eval.yaml` Rubric: `docs/superpowers/evals/tool_quality_rubric.yaml` Overview: `docs/superpowers/evals/README.md`

Pass threshold

**≥80/100** weighted; ship at **≥90** or accepted `pass_with_warnings`.

| Dimension | Weight | |-----------|--------| | connectivity | 20 | | schema_validity | 10 | | handler_correctness | 20 | | capture_quality | 10 | | visual_fidelity | 10 | | webmcp_parity | 10 | | intentcall_authoring | 10 | | docs_truth | 10 |

Iteration workflow

Task Progress:
- [ ] Start app (web-showcase or make showcase)
- [ ] Fresh WS_URI from log (not stale after hot reload)
- [ ] Run battery
- [ ] Record score + warnings in tracker
- [ ] Promote recurring patterns if repeated ≥2 times

Web iteration

make web-showcase
export WS_URI="$(grep -Eo 'ws://127\.0\.0\.1:[0-9]+/[A-Za-z0-9_=-]+/ws' .showcase/web_app.log | tail -1)"
bash tool/evals/run_dogfood_eval.sh --ws-uri "$WS_URI" --merge --skip-visual   # yq or dart fallback
# or without merge:
bash tool/evals/run_dogfood_eval.sh --ws-uri "$WS_URI"

Auto-runs: static gates + `validate-runtime` + `webmcp verify`.

Static only (CI / PR)

make dogfood-eval-static

Makefile

make dogfood-eval          # needs WS_URI in env
make dogfood-eval-static

Artifacts

• Per run: `.showcase/eval_runs/<timestamp>/eval_run.yaml`
• Snapshot: `.showcase/eval_run_<timestamp>.yaml`
• With `--merge`: updates `docs/evidence/dogfood/dogfood_web_eval.yaml` (`yq` or `dart run mcp_server_dart/tool/merge_dogfood_tracker.dart`)

Verdicts

| Verdict | Meaning | |---------|---------| | `pass` | score ≥80, no blocking warnings | | `pass_with_warnings` | score ≥80, e.g. `visual_capture_truth_mode` | | `fail` | score <80 | | `blocked_no_runtime` | `--skip-runtime` only |

Exec vs MCP names

Document in iteration notes when testing invoke:

• **exec:** bare names (`get_recent_logs`, `dogfood_ping`) — also accepts `fmt_*` aliases
• **MCP:** `fmt_` prefix (`fmt_get_recent_logs`, `fmt_client_tool` + listing `name`)
• **Connection:** global `--vm-service-uri` (not `exec --target`)

dart run mcp_server_dart/bin/flutter_mcp_toolkit.dart \
  --vm-service-uri "$WS_URI" exec --name get_recent_logs --args '{}'

Chrome battery notes

• Skip the visual track (`--skip-visual`): its fixture home (external `flutter_harness`) is retired, so the visual evidence is paused, not regenerable
• `validate-runtime --save-images` can hang >5m on Chrome; battery omits it unless `DOGFOOD_SAVE_IMAGES=1`

CI (branch)

`.github/workflows/intentcall_eval.yml` — PR: `dogfood-eval-static`; weekly/main: intentcall package tests + static.

Full Chrome runtime dogfood stays **local** until headless WebMCP is cost-effective.

After each iteration

1. Diff `dimension_scores` vs previous iteration in tracker. 2. Add to `recurring_warnings` / `recurring_errors` when repeated. 3. Update `fix_recommendations` with actionable doc/code deltas. 4. Do **not** treat tracker YAML as gate without a fresh battery run.

Related skills

• `flutter-mcp-toolkit-maintain-web` — WebMCP launch + verify
• `flutter-mcp-toolkit-maintain-macos` — macOS showcase
• `flutter-mcp-toolkit-repo-maintainer` — `make sync-skills` after editing this skill