/opik-annotate

name: opik-annotate
description: Set up human review in Opik — create an annotation queue for a project's traces or threads and give it a custom annotator view, an HTML page that shows annotators each item the way they need to judge it (the email beside the reply, the conversation as chat bubbles) instead of raw JSON. Writes through the Opik MCP when connected, the Opik SDK otherwise. Returns the queue, its link, and what annotators will see. Use for "build a custom annotator view", "set up an annotation queue", "make the review queue show the email and the reply side by side", "change what annotators see", "remove the custom view". Not for automatic scoring (use online-eval) or for finding which traces to review (use diagnose).
compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires Opik configured and a project with traces. Custom annotator views need an Opik version that supports them; an older Opik accepts the view and drops it, and the MCP says so. Without the MCP, it needs the Opik SDK `opik>=2.2.122`.
allowed-tools:
  - Read
  - Grep
  - Glob
  - Bash
metadata:
  last_updated: "2026-10-28"
  source_commit: "e4a6fa9b424128eb01817b8700761ac249a89e78"
  argument-hint: "[queue name or id, or what annotators should judge; optional project]"

Annotate — A Review Queue Annotators Can Read

**Definition of done:** the **queue exists with the view saved, and the user knows where to check it** — the write returned without a `warning`, and the answer carries the queue's link. A view is built from the project's **real** traces or threads, never from a guessed shape. If the queue or the view can't be saved, stop at the **first** genuine blocker and return **exactly one** next step.

Operate: **look at real items first, cover every shape you saw, show text not JSON, keep scoring in Opik's own panel.** This skill writes to Opik only and changes no application code.

Inputs

`/opik-annotate <what annotators judge>` ("the drafted reply", "whether the agent solved the ticket"), `/opik-annotate <queue name or id>` (change that queue's view), or a request to remove a view. Infer the rest; optional overrides: project (default: configured) · scope (default: `trace`; `thread` for whole conversations) · feedback definitions (default: the queue's own).

Activation — the only in-scope work

1. Resolve the project and the queue

With the Opik MCP connected: `list('annotation_queue', project_name=…)` shows the project's queues with their scope and item count; `read('annotation_queue', id_or_name)` returns one with its instructions, feedback definitions and view summary. The view's html is stated by size; `read('annotation_queue', id, fields=['annotator_view.html'])` returns it whole — read it before changing a view, and change only what was asked.

2. Look at real items before writing anything

• Queue with items: `list('trace', project_name=…, filters='annotation_queue_ids contains "<queue_id>"', size=20)` (`list('thread', …)` for a thread queue), then `read('trace', id)` on two or three that differ.
• Empty or new queue: the project's latest traces or threads instead, and say so ("the queue has no items yet, so the view is built from the latest traces").

Note every shape `input`, `output` and `metadata` take: objects, arrays, or strings that hold JSON. A view that handles only the most common shape fails annotators on the rest.

3. Decide what annotators need

From the queue's instructions and feedback definitions: what is judged goes first; what it is judged against goes beside or below it; what would bias the annotator (judge scores, when they give their own verdict) goes behind `kit.revealGate`.

4. Write the view

Read `references/kit-v1.md` first: it is the only API, class and token list the page has, and views are stored with that kit version. What the page can rely on:

• Rendered in a sandboxed frame: **no network** (no fetch, external script, stylesheet, font or image), no forms, popups or navigation. Inline `<script>` and `<style>` only.
• Build elements with `kit.h` and `text`; `innerHTML` and its relatives are blocked.
• Render through `kit.mount(render)` exactly once. `render` gets `{ trace }` for a trace queue and `{ thread, traces }` (turns in order) for a thread queue; if `render` throws or returns nothing, annotators get Opik's default view and see the error, so handle every shape you found instead of relying on that fallback.
• Trace fields: `id`, `name`, `input`, `output`, `metadata`, `tags`, `feedback_scores`, `start_time`, `end_time`, `duration`, `usage`, `total_estimated_cost`. Thread fields: `id`, `start_time`, `end_time`, `duration`, `number_of_messages`, `status`, `tags`, `feedback_scores`, `usage`, `total_estimated_cost`. No spans; attachments stored outside the trace are not available, media embedded as `data:` URLs is.
• `feedback_scores[]` with `source === "ui"` are human; the rest are judges.
• Style with `var(--kit-…)` tokens, never hard-coded colours: light and dark themes share the view. Layout must work from 360px wide.
• At most 200,000 characters.

Design: lead with what is judged in a `highlight` card; a row of `kit.chip` facts that frame the item; counts that matter (words against a limit); fold long text with `kit.clampGroup` and lists over ~5 items; nothing scrolls inside the view; technical details (model, ids, durations) in a collapsed `<details>` at the bottom. Normalise every shape into one before rendering. For a thread, `kit.turn` per message, user on the right, tool calls in a collapsed `<details>`.

5. Save it

MCP: `write('annotation_queue.create', {project_name, name, scope, instructions?, feedback_definition_names?, annotator_view: {html, summary}})`, or `write('annotation_queue.update', {id, annotator_view: {html, summary}})` for an existing queue. `summary` is one line starting "Annotators see…" describing the whole view; the queue form shows it as writt