/opik-optimize

name: opik-optimize
description: Improve a prompt with the Opik Agent Optimizer — resolve the prompt, a dataset, and a metric, pick the algorithm, run a bounded optimization, check the gain on held-out data, and save the winner as a new prompt version with the optimization run link. Runs via the opik-optimizer package; reads prompts and datasets via the MCP when connected. Use for "optimize this prompt", "improve my system prompt", "make the agent answer better", "tune the prompt against my dataset", "run the prompt optimizer". Not for measuring quality once (use evaluate), before/after on a suite (use compare), or hand-editing a prompt without data.
compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python project with Opik configured, a provider API key, and a dataset (or traces to build one). Install the `opik` skill alongside this one — it holds the shared dataset and prompt-library references; without it, this skill falls back to the public docs.
allowed-tools:
  - Read
  - Grep
  - Glob
  - Bash
  - Write
metadata:
  last_updated: "2026-09-15"
  source_commit: "2.0.0"
  argument-hint: "[prompt name or file, optional dataset and metric]"

Optimize — Improve a Prompt Against Data

**Definition of done:** an **optimized prompt** whose score on the metric **beats the baseline on data it was not tuned on**, the **optimization run link** in Opik, the cost of getting there, and the winner **saved as a new prompt version** (when the prompt lives in the library) — with the swap into code left as the next step. If the optimization can't run within a stated budget, stop at the **first** genuine blocker and return **exactly one** next step. A prompt that scores higher only on its own training items is not an improvement.

Operate: **measure the baseline first, state the budget before spending it, hold data out, pick the algorithm for the failure you see, save the winner where it can be versioned — and change no application code.** The only file this skill writes is a runner outside the repo; the prompt is saved to Opik, not into the codebase.

Inputs

The entry point is `/opik-optimize <prompt-name>` (a prompt-library prompt), `/opik-optimize <path or function>` (a prompt in code), or `/opik-optimize` (find the system prompt in this repo). Infer the rest; treat these as **optional overrides**:

• dataset (default: existing dataset for the project → export the regression suite → build from traces) · metric (default: heuristic on `expected_output` if present, else one binary judge) · algorithm (default: `MetaPromptOptimizer`) · budget (default: `n_samples=50`, `max_trials=10`) · model · validation split (default: hold out 20%).

Ask only at a genuine, non-inferable blocker (see **Blockers**).

Activation — the only in-scope work

1. Resolve the prompt

• **Library:** `client.get_chat_prompt(name)` (or `get_prompt` for a text prompt). Note the current `version` — that is the baseline.
• **Code:** grep for the system prompt / `messages=[...]`; read it verbatim. Note where it lives; you will not edit it.
• **Trace:** the `llm` span's `input.messages` on a representative trace.

The optimizer's `opik_optimizer.ChatPrompt` is a **different class** from the library's `opik.ChatPrompt` — build it from the raw messages yourself: `references/sdk-snippets.md` (**Resolve the prompt**).

2. Resolve the dataset (and hold some out)

The optimizer needs an `opik.Dataset` whose item keys match the prompt's `{variables}`: `references/sdk-snippets.md` (**Resolve the dataset**).

• Only a **test suite** exists → export its items into a dataset once: `suite.get_items()` → `client.get_or_create_dataset("<suite>-optimize", project_name=…)` → `insert([{**it["data"]} …])`.
• Nothing exists → build from traces (`search_traces` → `{"question": t.input[...], "expected_output": …}`) or run `/opik-evaluate` first.
• **Hold out:** split into train and validation datasets and pass `validation_dataset=`. Note what it does: the optimizer **scores every trial on `validation_dataset`** and uses the train set to show the reasoning model examples — so the validation set is the selection set, and it needs ≥10 items or every candidate ties (a 4-item split logs `n_samples … larger than evaluation dataset size` and cannot separate prompts). If you need a gain measured on items the optimizer never saw, keep a third split and re-score the winner on it with `evaluate()`. Fewer than ~20 items in total → say the result will be noisy; below 10 → **Blocker**.

3. Define the metric

A function `(dataset_item, llm_output) -> float`, higher is better. **Give it a real name** (`def refund_answer_similarity(...)`) — its `__name__` becomes the Optimization run's objective name in the UI and `result.metric_name`; a function called `metric` shows up as "metric".

• `expected_output` present → heuristic (`LevenshteinRatio`, `Equals`, or a task-specific check) — deterministic and free. Prefer a **graded** metric over exact match: when the baseline scores 0.0 on every item (observed with `Equals` on a strict output format), every candidate also scores 0.0 and the optimizer has nothing to climb — five trials of flat zeros is a metric problem, not a prompt problem.
• Otherwise → **one** binary judge for the failure mode being optimized (`../opik-evaluate/references/write-judge-prompt.md`), wrapped to return its score `.value`. Multi-objective → `MultiMetricObjective`.

Never optimize against a judge nobody validated: an unvalidated judge is the easiest thing to overfit.

4. Pick the algorithm

| Failure you see | Optimizer | |---|---| | Instructions unclear / underspecified (general default) | `MetaPromptOptimizer` | | The model needs examples of the right answer; few-shot is acceptable | `FewShotBayesianOptimizer` | | Failures cluster into a few root causes | `HierarchicalReflectiveOptimizer` (`HRPO`) | | Larger budget, want broad search | `Evolutionary