/opik-online-eval

name: opik-online-eval
description: Take a judge live on production traffic — create an Opik online evaluation rule (LLM-as-judge or Python metric) on a project with sampling, filters, variable mapping, and a cost cap, then confirm new traces are being scored. Works over the SDK's REST client; reads rules and score names via the MCP when connected. Returns the rule, the score name it emits, and how to watch it. Use for "score production traces", "monitor hallucinations in prod", "take this judge live", "set up an online evaluation rule", "alert me when quality drops". Not for offline experiments (use evaluate or compare) or for finding what is already broken (use diagnose).
compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires Opik configured and a project that receives traces. Install the `opik` skill alongside this one — it holds the shared production and observability references; without it, this skill falls back to the public docs.
allowed-tools:
  - Read
  - Grep
  - Glob
  - Bash
metadata:
  last_updated: "2026-09-15"
  source_commit: "2.0.0"
  argument-hint: "[what to score, or a judge from the evaluate skill; optional project]"

Online Eval — Take a Judge Live

**Definition of done:** an **online evaluation rule exists on the project, is enabled, and has scored at least one new trace** — confirmed by reading a fresh trace's feedback score, not by the create call returning. The rule scores one failure mode, samples at a rate the project's volume can afford, carries a cost cap, and maps its variables to the fields the traces actually have. If traffic hasn't arrived yet, the done state is "rule live, unverified — watch this filter". If the rule can't be created, stop at the **first** genuine blocker and return **exactly one** next step.

Operate: **one failure mode per rule, sample before you scale, cap the spend, verify on a real trace — and change no application code.** This skill writes to Opik only.

Inputs

The entry point is `/opik-online-eval <what to score>` ("hallucinations", "the refund-window answer", "refusals"), `/opik-online-eval` right after `/opik-evaluate` validated a judge (take that judge live), or `/opik-online-eval <project>`. Infer the rest; treat these as **optional overrides**:

• project (default: configured) · scope (default: **trace**; `span` for one model call, `thread` for whole conversations) · sampling rate (default: 1.0 under ~1k traces/day, else 0.1–0.2) · filters (default: none; typical: an `environment` or tag filter) · judge model · cost cap (default: set one) · variable mapping (default: `input → input`, `output → output`).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

Activation — the only in-scope work

1. Resolve project, judge, and scope

Confirm Opik is reachable (`~/.opik.config` or `OPIK_API_KEY`; otherwise → **Blocker**). Resolve the project id from a trace or by name. Decide the judge:

• A judge `/opik-evaluate` already validated → reuse its prompt and output schema verbatim.
• A named metric (hallucination, answer relevance, moderation) → a minimal binary judge for that one failure mode (`../opik-evaluate/references/write-judge-prompt.md`).
• A mechanical check (JSON valid, contains a disclaimer, latency budget) → a **Python metric rule**, not a judge.

Scope: `trace` by default; `span` when the check is about one LLM/tool call; `thread` when the check needs the whole conversation (thread rules wait for the thread to go inactive — 15 min by default).

2. Look at the traces before mapping variables

Read three recent **production** traces and note the real shape of `input` and `output`. Experiment runs land in the same project with a different input shape (their metadata carries `test_suite_experiment_id`), so skip those — filtering on the app's entrypoint is the reliable way: `client.search_traces(project_name=…, max_results=3, filter_string='name = "<entrypoint>"')`. (OQL has no `is_empty` for `metadata.*` keys.) Variables are **plain field paths**, dot-notation for nested keys (`output.answer`, `input.messages`) — never `{{ }}` templates. A wrong path is the most common reason a rule silently scores nothing.

3. Check what already runs

import opik

client = opik.Opik()
existing = client.rest_client.automation_rule_evaluators.find_evaluators(project_id="<project_id>")

Same name or same failure mode already there → don't create a second one; report `exists` (offer to adjust sampling/enable). When the hosted MCP is connected, `list('online_rule', project_id=…)` and `list('score_name', project_id=…)` show the same, with each rule's type, enabled flag, and sampling rate.

4. Create the rule

There is **no high-level SDK wrapper**; use the REST client. LLM-as-judge, trace scope:

from opik.rest_api.types import (
    AutomationRuleEvaluatorWrite_LlmAsJudge,
    LlmAsJudgeCodeWrite,
    LlmAsJudgeModelParametersWrite,
    LlmAsJudgeMessageWrite,
    LlmAsJudgeOutputSchemaWrite,
)

rule = AutomationRuleEvaluatorWrite_LlmAsJudge(
    action="evaluator",  # required literal; the model rejects the payload without it
    name="refund_window_correct",  # becomes the feedback-score name on every scored trace — use underscores, not hyphens: OQL parses `feedback_scores.a-b` as an operator
    project_ids=["<project_id>"],
    sampling_rate=0.2,  # fraction of SDK-logged traces scored
    enabled=True,
    filters=[],  # e.g. [{"field": "tags", "operator": "contains", "value": "production"}]
    code=LlmAsJudgeCodeWrite(
        model=LlmAsJudgeModelParametersWrite(name="<judge model>", temperature=0.0),
        messages=[
            LlmAsJudgeMessageWrite(
                role="USER", content="<the validated judge prompt using {{input}} and {{output}}>"
            )
        ],
        variables={"input": "input", "output": "output"},  # field paths from step 2
        schema_=[
            LlmAsJudgeOutpu