/opik-test

name: opik-test
description: Turn a failing Opik trace (or a described failure) into a repeatable regression check — a test-suite item with the trace's input and one or two binary assertions — so a fix can be verified by the compare skill. Works over the SDK; uses the MCP write tool when connected. Returns the suite, the item, and the assertion. Use for "turn this into a test", "add a regression case for this trace", "make sure this doesn't happen again", "capture this failure", "add this to the test suite". Not for running the suite (use compare), building an evaluation from scratch (use evaluate), or explaining why the trace failed (use explain).
compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured and at least one trace, or a described input/expected pair. Install the `opik` skill alongside this one — it holds the shared test-suite and dataset references; without it, this skill falls back to the public docs.
allowed-tools:
  - Read
  - Grep
  - Glob
  - Bash
metadata:
  last_updated: "2026-09-15"
  source_commit: "2.0.0"
  argument-hint: "[trace id, or a description of the failure to capture]"

Test — Capture a Failing Case as a Repeatable Check

**Definition of done:** one test-suite item exists in Opik that reproduces the failure — its `data` carries the failing input (and the expected output when one is known), and it has **one or two binary assertions** that would have failed on the bad trace and pass on a correct one. The item is in a named suite scoped to the project, is confirmed by reading it back, and is ready for `/opik-compare` to run. If the case can't be captured, stop at the **first** genuine blocker and return **exactly one** next step. Writing a pytest file, or describing a test in prose, is not success.

Operate: **extract the case from real trace data, write assertions a judge can decide with a yes or no, store them where `/opik-compare` will find them — and change no application code.** This skill writes to Opik, never to the repo.

Inputs

The entry point is `/opik-test <trace-id>` (capture that trace), `/opik-test <describe the failure>` (find the trace, or capture from the description alone), or `/opik-test` right after `/opik-explain` (capture the trace it just explained). Infer the rest; treat these as **optional overrides**:

• suite name (default: `<project>-regressions`) · project (default: the trace's project, else the configured one) · expected output (default: none — assertions carry the expectation) · execution policy (default: the suite's; `runs_per_item: 3, pass_threshold: 2` only for an intermittent failure).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

Activation — the only in-scope work

1. Resolve the case

• A **trace id** (uuid-shaped): that trace is the case.
• A **description** ("the refund answer is wrong", "it hallucinated the shipping time"): `search_traces` on the project for the matching trace (error, low score, or free-text match) and take the best one. If none matches, capture from the description alone — the user's input and expectation become the item.
• Confirm Opik is reachable: if `~/.opik.config` exists or `OPIK_API_KEY` is set, use it. Otherwise → **Blocker** ("run `opik configure`, then rerun").

2. Read the trace (SDK-first)

import opik
client = opik.Opik()

tid = "<trace_id>"
trace = client.get_trace_content(tid)       # TracePublic: .input, .output, .project_id — NOT .project_name (accessing it raises)
project = client.rest_client.projects.get_project_by_id(trace.project_id).name
spans = client.search_spans(project_name=project, trace_id=tid)   # root span (no parent_span_id) names the entrypoint the compare skill will call
# Pass project_name: without it search_spans looks in the configured default project and returns nothing.

Take the **input** exactly as the trace recorded it (the root span / trace `input`), the **actual output** (what went wrong), and the **root span name** (the entrypoint). When the MCP is connected, `read('trace', id)` is an equivalent path — a convenience, not a requirement.

3. Write the assertions

Assertions are plain-English statements an LLM judge answers **pass/fail** from the item's `data` and the candidate output. Rules: 1. **One failure mode per assertion.** State what a correct output does, not a list of qualities. 2. **Would have failed on the bad trace.** Check it against the actual output you just read; if it would pass, it's the wrong assertion. 3. **Decidable from the output alone** (plus `expected_output` when present). No "is helpful", no Likert scales. 4. **At most two**: the positive expectation, and — only if the bad output did something specific and wrong — one negative ("does not claim …"). Prefer the positive form: small judge models misread negatives (observed: "does not promise a refund within 24 hours" judged *true* on an output that promised exactly that). If the negative matters, fold it into the positive ("states 5-7 business days, not 24 hours").

Prefer a deterministic check over a judge when the correct answer is exact: put it in `data.expected_output` as well, so `/opik-compare` can score it with a heuristic metric.

4. Store the case (create or append)

Suite naming: `<project>-regressions` unless the user names one. Reuse an existing suite; never create a second suite for the same project.

suite = client.get_or_create_test_suite(
    name="<project>-regressions",
    project_name=project,
    tags=["regression"],
)

# Dedupe on the source trace before inserting.
existing = suite.get_items(filter_string=f'data.source_trace_id = "{tid}"')
if not existing:
    suite.insert([{
        "data": {
            "input": trace.input,                 # verbatim from the trace
            "source_trace_id": tid,
            # "expected_output": "...",           # only when the corre