/opik-evaluate

name: opik-evaluate
description: Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link. Picks a test suite with judge assertions or a dataset with metrics, sources cases from traces or synthetic data, scores heuristics-first then one-failure-mode judges, runs client-side via the SDK or server-side for prompt-only targets, and reads the scores back. Covers RAG evaluation, error analysis, writing and validating LLM judges against human labels, and auditing an existing eval pipeline. Use for "evaluate my agent", "measure quality", "build an eval", "write an LLM judge for hallucinations", "audit our evaluation pipeline", "how good is my RAG", "set up evals for this". Not for before/after on an existing suite (use compare), one regression case (use test), scoring production traffic (use online-eval), or the ship/hold decision (use verify).
compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured. Install the `opik` skill alongside this one — it holds the shared test-suite, dataset, and metric references; without it, this skill falls back to the public docs.
allowed-tools:
  - Read
  - Grep
  - Glob
  - Bash
  - Write
metadata:
  last_updated: "2026-09-15"
  source_commit: "2.0.0"
  argument-hint: "[optional: what to evaluate, or a dataset/suite name]"

Evaluate — Build an Evaluation and Run It

**Definition of done:** an **experiment in Opik with scores**, its **link**, and a **score summary** the user can act on — produced by running the user's app (or prompt) over a set of cases with scoring that targets real failure modes. If the evaluation can't be built or run, stop at the **first** genuine blocker and return **exactly one** next step. A dataset with no run, a judge prompt with no scores, or a plan in prose is not success.

Operate: **ground the cases in what actually goes wrong, score with code before judges, judge one failure mode at a time, run once end to end, read the scores back from Opik — and change no application code.** The only file this skill writes is a runner outside the repo.

Inputs

The entry point is `/opik-evaluate` (evaluate the app in this repo), `/opik-evaluate <what>` ("the RAG answers", "the refund flow"), or `/opik-evaluate <dataset-or-suite>` (run against existing cases). Infer the rest; treat these as **optional overrides**:

• cases source (default: existing suite/dataset → recent traces → synthetic) · scoring (default: heuristics where an expected output exists, else one judge per failure mode) · approach (default: **test suite** for agents; **dataset + `evaluate()`** when you need built-in metrics or exact matching) · judge model · sample size (default: 20–50 items).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

Activation — the only in-scope work

1. Resolve the target

Confirm Opik is reachable (`~/.opik.config` or `OPIK_API_KEY`; otherwise → **Blocker**). Find the entrypoint to evaluate (the function a trace's root span names, or the one the user points at). Check what already exists — `client.get_test_suites(project_name=…)`, `client.get_datasets()` — and reuse before creating.

**Have an eval already?** Audit it first (`references/eval-audit.md`): unvalidated judges, no error analysis, vanity metrics. Fix the worst gap, then run.

2. Ground it in failures (error analysis)

Read real traces before writing any scorer — `client.search_traces(project_name=…, max_results=100)` (errors, low scores, long durations first). Categorize what goes wrong and how often (`references/error-analysis.md`). No traces yet → generate cases (`references/generate-synthetic-data.md`) and say so in the report.

3. Choose the shape

| Situation | Use | |---|---| | Agent / chatbot, expectations are behaviors ("mentions Paris", "declines legal advice") | **Test suite** — items + string assertions checked by a judge; `opik.run_tests()` | | Exact expected outputs, or built-in metrics (Hallucination, AnswerRelevance, RAG `ContextPrecision`/`ContextRecall`) | **Dataset + `evaluate()`** with `scoring_metrics` | | The thing under test is a prompt version, not code | **Server-side**: `client.rest_client.experiments.execute_experiment(...)` — no runner |

Prefer the test suite for agents; it is what `/opik-test` and `/opik-compare` operate on.

4. Build the cases

import opik
client = opik.Opik()

suite = client.get_or_create_test_suite(
    name="<project>-eval", project_name="<project>",
    global_assertions=["<one behavior every answer must show>"],   # optional
)
suite.insert([
    {"data": {"input": t.input, "source_trace_id": t.id}, "assertions": ["<what a correct output does>"]}
    for t in sampled_traces
])

For the dataset path: `client.get_or_create_dataset(name, project_name)` then `dataset.insert([{"input": …, "expected_output": …}])`. Store inputs verbatim; keep `source_trace_id` so cases trace back to production.

5. Define the scoring

1. **Code first.** `Equals`, `Contains`, `RegexMatch`, `IsJson`, `JsonSchemaMatch`, `LevenshteinRatio` from `opik.evaluation.metrics` whenever the check is mechanical. 2. **Then one judge per failure mode**, binary pass/fail, from step 2's categories (`references/write-judge-prompt.md`). As a suite assertion, or as a `GEval` / custom `BaseMetric` on the dataset path. 3. **RAG:** score retrieval and generation separately (`references/evaluate-rag.md`). 4. Before trusting a judge on anything that matters, calibrate it against a few human labels (`references/validate-evaluator.md`) — TPR/TNR, not accuracy.

6. Run it

Write the task adapter as a temp file **outside the repo** (needs the app's provider credentials — absent → **Blocker**). Never run a production entrypoint that writes, sends, or spends.

# Test suite
results = opik.run_tests(test_suite=suite, task=lambda item: {"input": item["input"