/opik-compare

name: opik-compare
description: Run a candidate against the baseline over an Opik test suite and read the numbers back — which cases broke, which got fixed, the per-metric deltas, worst rows, and whether the two runs are comparable — with the Opik compare-view link. Runs via the SDK; reads results via the MCP when connected. Does not issue a ship/no-ship verdict. Use for "did my fix work", "compare against the baseline", "run the regression suite", "why did quality drop", "which cases regressed", "compare these two experiments". Not for the ship/hold decision itself (use verify), live production triage (use diagnose), building an evaluation from scratch (use evaluate), or capturing a single case (use test).
compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured and a test suite (from the test or evaluate skill) or two existing experiments. Install the `opik` skill alongside this one — it holds the shared test-suite and experiment references; without it, this skill falls back to the public docs.
allowed-tools:
  - Read
  - Grep
  - Glob
  - Bash
  - Write
metadata:
  last_updated: "2026-09-15"
  source_commit: "2.0.0"
  argument-hint: "[suite name, or two experiment ids/names]"

Compare — Candidate vs Baseline Over a Suite

**Definition of done:** two experiments on the same suite — a **baseline** and a **candidate** — read back item by item, with the **per-metric deltas**, the cases that went **pass → fail** (regressions) and **fail → pass** (fixes), the worst rows, a note on whether the runs are comparable, and the **Opik compare-view link**. If only one run exists, the done state is "baseline created — rerun after the change". If the suite can't be run or read, stop at the **first** genuine blocker and return **exactly one** next step. Aggregate scores alone are not a comparison; a verdict is not this skill's job.

Operate: **run the candidate the same way the baseline was run, read both back from Opik rather than from the run's console output, name the specific cases that changed — and change no application code.** The only file this skill writes is a throwaway runner outside the repo.

Inputs

The entry point is `/opik-compare <suite>` (run the suite now as the candidate, compare against the latest prior run), `/opik-compare <experiment-A> <experiment-B>` (read two existing runs, no new run), or `/opik-compare` right after `/opik-test` (that suite). Infer the rest; treat these as **optional overrides**:

• suite (default: `<project>-regressions`, or the one `/opik-test` just wrote to) · baseline (default: the most recent earlier experiment on the suite) · candidate name (default: `candidate-<short git sha>`) · judge model for assertions (default: the suite's / the baseline's) · runs per item (default: the suite's policy).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

Activation — the only in-scope work

1. Resolve the suite and the baseline

The SDK calls: `references/sdk-snippets.md` (**Resolve the suite and the baseline**). Newest first is not guaranteed — sort the prior experiments by `created_at` yourself. Baseline = the most recent prior experiment on this suite, unless the user names one. **No prior experiment → this run *is* the baseline** (step 3 still runs; status `baseline_created`). Two explicit experiments → skip step 3, go to step 4.

Confirm Opik is reachable: if `~/.opik.config` exists or `OPIK_API_KEY` is set, use it. Otherwise → **Blocker**.

2. Build the task adapter (code candidates)

The suite's items say what to call: each item `description` written by `/opik-test` ends in `Entrypoint: <root span name>`. Grep the repo for that function, import it, and wrap it:

def task(item: dict) -> dict:
    return {"input": item["input"], "output": str(entrypoint(item["input"]))}

Write the runner as a **temp file outside the repo** (or a scratch path the user names) — never into the codebase, never committed. It needs the app's provider credentials; if they're absent → **Blocker** (the app can't answer the items). Never point the runner at a production entrypoint that writes, sends, or spends.

**Prompt candidates** (the change is a prompt version, not code): there is no adapter — run server-side instead: `references/sdk-snippets.md` (**Prompt candidates**).

3. Run the candidate

Same suite, same version, same judge model, same runs-per-item as the baseline — vary **only** the thing under test. The `run_tests` call, and the guard for a judge that fails without raising: `references/sdk-snippets.md` (**Run the candidate**). Pass `generate_report=False`: the default writes `opik_test_suite_reports/` into the user's repo. Do not read scores off `result` and stop — step 4 reads both runs from Opik so baseline and candidate go through the same path.

4. Read both runs back (SDK-first, MCP when connected)

The SDK read of both runs, and the side-by-side join: `references/sdk-snippets.md` (**Read both runs back**). `get_experiment_by_name` is deprecated — resolve names with `get_experiments_by_name` and pick by id. When the MCP is connected it does the whole of this step in one call: `list('dataset_item', experiment_ids=['<baseline_id>', '<candidate_id>'])` returns the cases side by side with each run's score, the per-case delta, the worst trace to open, and a warning when the two runs covered different cases or different dataset versions. Prefer it over the SDK join in `references/sdk-snippets.md`, which stays here for runs the MCP cannot reach. `list('experiment', …)` and `read('experiment', id)` still find and sanity-check the two runs.

5. Answer the questions, in this order

1. **Comparable?** Same suite version (`dataset_version` / item count), same judge model, same runs-per-item. If not, say so first — the deltas below are then indicative, not measured. 2. **Which cases broke** — `passed`