/opik-verify

name: opik-verify
description: Decide ship or hold for a candidate from the compare skill's numbers, against an explicit release policy — regressions, pass rate, safety-tagged cases, subgroup consistency, latency and cost budgets, flakiness, evidence size, and whether the judge is validated. Reads two experiments on an Opik test suite via the SDK (or the MCP when connected) and returns a verdict with every criterion shown pass/fail. Use for "is this safe to ship", "can I merge this", "go/no-go on this change", "gate this release", "should I roll this out". Not for producing the numbers (use compare), building an evaluation (use evaluate), or deploying anything.
compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires Opik configured and a test suite with a baseline and a candidate experiment (from the compare skill). Install the `opik` skill alongside this one — it holds the shared test-suite and experiment references; without it, this skill falls back to the public docs.
allowed-tools:
  - Read
  - Grep
  - Glob
  - Bash
  - Write
metadata:
  last_updated: "2026-09-17"
  source_commit: "2.0.0"
  argument-hint: "[suite, or baseline and candidate experiment ids; optional --policy path]"

Verify — Ship or Hold, Against a Policy You Can Read

**Definition of done:** one verdict — **`ship`**, **`hold`**, **`needs_review`**, or **`insufficient_evidence`** — computed from a **declared policy** over the baseline-vs-candidate numbers, with **every criterion listed with its threshold, the observed value, and pass/fail**, the cases behind any failure named, and the compare-view link. The policy is either the repo's `opik-release-policy.yaml` or the documented defaults, and the report says which. If the two runs can't be read or aren't comparable, stop at the **first** genuine blocker and return **exactly one** next step. "Looks good to me" is not a verdict; a verdict without its criteria is not one either.

Operate: **apply the policy mechanically, show your arithmetic, refuse to ship on a judge nobody validated, and change no application code.** The only file this skill may write is the policy file, and only when the user says so. It never deploys.

Inputs

The entry point is `/opik-verify` right after `/opik-compare` (its baseline and candidate), `/opik-verify <suite>` (the two most recent runs on the suite), or `/opik-verify <baseline-id> <candidate-id>`. Infer the rest; treat these as **optional overrides**:

• policy (default: `opik-release-policy.yaml` at the repo root or under `.opik/`, else the defaults below) · which experiments (default: as above) · `--record` (default: off — write the verdict into the candidate experiment's config).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

The policy

Every key is optional; missing keys take these defaults. Say in the report which source applied.

# opik-release-policy.yaml — repo root or .opik/. Versioned with the code so the gate is reproducible.
min_items: 10                      # fewer scored items than this -> insufficient_evidence, never ship
max_regressions: 0                 # pass -> fail cases allowed (flaky items excluded when flaky_policy: exclude)
pass_rate: not_below_baseline      # or a number in 0..1; candidate pass rate must satisfy it
safety_tags: [safety]              # a regression on an item whose data.tags contains one of these -> hold, always
subgroup_key: null                 # a data key (e.g. "category"); no subgroup's pass rate may fall
latency_p90_max_increase: 0.25     # candidate p90 duration vs baseline (experiments expose p50/p90/p99)
cost_per_item_max_increase: 0.25   # candidate mean cost per item vs baseline, as a fraction
flaky_policy: exclude              # exclude | count — an item that flips between runs of the SAME code is flaky
judge_validated: false             # set true once the suite's judge has been checked against human labels

`judge_validated: false` is the **human-review gate**: until someone has confirmed the judge agrees with people (`/opik-evaluate`'s `validate-evaluator` reference), a passing run yields `needs_review`, not `ship`. Flip it to `true` in the file once that is done — deliberately a human edit, never something this skill sets on its own.

Activation — the only in-scope work

1. Load the policy

Look for `opik-release-policy.yaml` at the repo root, then `.opik/`. Parse it; unknown keys → **Blocker** (name the key). No file → defaults, and say so. Never invent thresholds not in the file or the defaults.

2. Resolve the two runs

Take them from `/opik-compare`'s output when it just ran. Otherwise:

import opik

client = opik.Opik()
runs = sorted(
    client.get_test_suite_experiments(name="<suite>", project_name="<project>"),
    key=lambda e: e.get_experiment_data().created_at,
)
baseline, candidate = runs[-2], runs[-1]  # or the two ids the user gave

Skip a **failed-judge run** (a run whose judge had no credential is not a candidate — `/opik-compare` explains how it happens). `scoring_failed` does not survive the read path; the read-back signal is: **every item failed and every assertion `reason` mentions a missing credential or an LLM infrastructure error**. Say which run you skipped and why. When the hosted MCP is connected, `list('experiment', name=…)` shows each run's averages and pass rate to pick from; the item-level read below stays on the SDK.

3. Read both runs, item by item

An experiment holds **one item per run**: with `runs_per_item: 3` a dataset item appears three times, same `dataset_item_id`, different `trace_id`. Group — a dict keyed on `dataset_item_id` silently keeps one run and loses the counts.

from collections import defaultdict


def by_item(exp):
    groups = defaultdict(list)
    for i in exp.get_items():
        groups[i.dataset_item_id].append(i)  # each: dataset_item_data (tags / subgroup key),
    return