/skill-compare

Skill Comparison Experiment

Run an apples-to-apples A/B comparison between two refs of the skills repo — each ref is either a branch or a commit SHA — and produce a decision-ready report. Automates the workflow described in [`tests/experiments/skill-comparison-playbook.md`](../../tests/experiments/skill-comparison-playbook.md).

**Input:** `$ARGUMENTS`

• `<ref_a> <ref_b>` — required, the two refs to compare. Each ref is either a **branch name** (local or remote-tracking) or a **commit SHA** (short or full, must already be in the local object database). The two refs can be any mix of the two kinds.
• `<task_selector>` — optional third argument. Restricts which tasks run. Defaults to **all tasks** (warn the user: this can be slow and expensive). Accepts three forms; pick the most specific one that fits the question:
• **Skill name** (bare word): `uipath-maestro-flow` → all task files under `tests/tasks/uipath-maestro-flow/`. Use this when the comparison is scoped to one skill.
• **Tag filter** with `tags:` prefix: `tags:smoke` or `tags:smoke,init` → comma-separated tags, OR semantics (any task carrying any of these tags). Forwarded to `coder-eval run --tags`. Use this for cross-skill slices like "only smoke tests" or "only e2e + connector tests".
• **Path glob** with `paths:` prefix: `paths:tasks/uipath-maestro-flow/smoke/init_validate.yaml` or `paths:tasks/*/smoke_*.yaml` (comma-separated globs allowed). Use this when you want a hand-picked subset.
• `<n_reps>` — optional fourth argument. Reps per variant. Defaults to `3`. Accept integers 1–5.

**Examples:**

• `/skill-compare main feat/my-change uipath-maestro-flow` — two branches, scoped to one skill.
• `/skill-compare main a1b2c3d uipath-maestro-flow` — branch vs. commit SHA, scoped to one skill.
• `/skill-compare a1b2c3d e4f5g6h uipath-maestro-flow` — two SHAs (compare two historical points).
• `/skill-compare main feat/my-change tags:smoke` — two branches, only smoke tests across all skills.
• `/skill-compare main feat/my-change tags:smoke,init 5` — smoke OR init tasks, N=5.
• `/skill-compare main feat/my-change paths:tasks/uipath-maestro-flow/smoke/init_validate.yaml,tasks/uipath-maestro-flow/smoke/registry_discovery.yaml` — two specific files.
• `/skill-compare main feat/my-change` — all tasks across all skills, N=3 (slow + expensive; expect to be re-prompted at Phase 5).

**Ref slugs** (used in filenames and worktree paths):

• Branch: replace `/` with `-` and strip any leading `origin/`. Example: `feat/my-change` → `feat-my-change`.
• SHA: resolve to the 7-char short form via `git rev-parse --short`. Example: `a1b2c3defabc` → `a1b2c3d`. Short SHAs are already filename-safe; no further transformation.

**Output:**

• A new experiment YAML at `tests/experiments/compare-<ref_a_slug>-vs-<ref_b_slug>.yaml`
• Worktrees at `../skills-<ref_a_slug>` and `../skills-<ref_b_slug>` relative to the repo root
• Run results under `tests/runs/compare-<ref_a_slug>-vs-<ref_b_slug>-<timestamp>/`
• Comparison report at `tests/runs/compare-<ref_a_slug>-vs-<ref_b_slug>-<timestamp>/comparison-report.md`

**Working directory.** Every phase runs from `tests/`. All git operations use `git -C ..` to target the repo root. Worktree paths like `../skills-<slug>` resolve alongside the repo, one level above `tests/`.

---

Phase 1 — Parse & validate

1. Parse `$ARGUMENTS`: require at least two non-empty tokens. If fewer than 2, print usage and stop. 2. For each ref, reject inputs that start with `-` — they can be misread as git flags. Stop with a usage message. 3. **Classify each ref as branch or SHA.** Use `--` to terminate flag parsing throughout:

   # Is it a branch (local or remote-tracking)?
   git -C .. show-ref --verify --quiet "refs/heads/<ref>"         && kind=branch     # local branch
   git -C .. show-ref --verify --quiet "refs/remotes/origin/<ref>" && kind=branch-remote  # remote-only
   # Otherwise, does it resolve to a commit in the local object database?
   [ "$(git -C .. cat-file -t "<ref>" 2>/dev/null)" = "commit" ]  && kind=sha

• If `kind=branch`: strip leading `origin/` from the recorded ref if present. Slug = branch name with `/` → `-`.
• If `kind=branch-remote`: record the remote-only flag — Phase 3 will need `-b <branch> origin/<branch>` to create a tracking branch. Slug = branch name with `/` → `-`.
• If `kind=sha`: resolve to the 7-char short form with `git -C .. rev-parse --short --verify -- "<ref>"`. Slug = the short SHA. Record the pinned SHA as the short form.
• If none match: stop and report the ref as not found. For SHAs that might be unfetched, hint at `git fetch origin` or `git fetch origin <sha>` (git supports fetching by SHA on most remotes).

4. Reject the call if the two refs resolve to the **same commit SHA** — comparing identical content produces no signal. Treat `main` and `a1b2c3d` as identical if `main` currently points at `a1b2c3d`; warn the user and stop. 5. Reject the call if `<n_reps>` is outside `1..5`. N=1 is allowed for a fast signal check but warn it's underpowered for decisions. 6. If `<task_selector>` is provided, classify and validate it:

• **`tags:` prefix** → split the rest on `,` and record as a tag list. No filesystem check at this stage; Phase 5 will warn if the tag list matches zero task files.
• **`paths:` prefix** → split the rest on `,` and record as a glob list (paths relative to `tests/`). For each glob, check it expands to at least one existing `.yaml` file under `tests/tasks/`. If any glob expands to zero files, stop and list which one was empty.
• **Bare word** (no prefix) → treat as a skill name. Verify `tasks/<word>/` exists (relative to `tests/`). If not, list available skills under `tasks/*/` and stop.
• **Anything else** (e.g. `tags:` with no list, `paths:` outside `tests/tasks/`) → stop with a usage message showing the three accepted forms.

7. For each ref, decide whether to create or reuse a worktree at `../skills-<slug>`:

• If the path d