/phoenix-harbor

name: phoenix-harbor
description: Configure and interpret the Phoenix plugin for Harbor agent evaluations. Use when adding `arize-phoenix` to Harbor jobs, choosing ATIF tracing, mapping Harbor tasks and rewards to Phoenix experiments, comparing agents or models, resuming jobs, or troubleshooting Harbor records in Phoenix.
license: Apache-2.0
metadata:
  targets: "agents mcp"
  author: oss@arize.com
  version: "1.0.0"

Phoenix for Harbor

Use the Phoenix Harbor plugin to record Harbor agent evaluations as versioned Phoenix datasets, experiments, runs, scores, and ATIF traces.

Harbor runs agents and verifiers. Phoenix records and compares their results. Do not describe Phoenix as executing Harbor tasks or recalculating Harbor rewards.

Requirements

• Python 3.12 or newer
• Harbor 0.21.0 or newer
• `arize-phoenix-client` installed with the `harbor` extra
• Phoenix server 15.0 or newer

Install the client and Harbor in the same Python environment:

uv pip install "arize-phoenix-client[harbor]"

Choose the trace mode

Use `atif` unless the agent has no ATIF trajectory or the user does not want traces. ATIF is the default. It reads trajectory files after the final trial attempt, so the sandbox needs no Phoenix endpoint, credentials, instrumentation, or outbound network access.

Use `null` to record datasets, experiments, runs, and evaluations without traces:

--plugin-kwarg trace_mode=null

Live OpenTelemetry Protocol (OTLP) support is deferred to a follow-up. This release accepts `atif` or `null`, and does not link live OpenTelemetry agent traces to experiment runs.

Set the Phoenix destination

Set the endpoint for the Phoenix plugin process. ATIF agents and their sandboxes do not connect to Phoenix. Prefer environment variables so credentials do not enter shell history or job configuration:

export PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006
export PHOENIX_API_KEY=your-api-key

Omit `PHOENIX_API_KEY` when the Phoenix instance does not require authentication. The `endpoint` and `api_key` plugin kwargs override these values when the user asks for per-job settings.

Add the plugin

Add `--plugin arize-phoenix` to the user's existing `harbor run` command. Preserve their dataset, agent, model, environment, concurrency, retry, and task selections.

harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a terminus-2 \
  -m openai/gpt-5-mini \
  --plugin arize-phoenix \
  --yes

Do not invent or replace Harbor settings that are unrelated to Phoenix.

Predict the Phoenix records

Use this mapping when explaining a job or checking its results:

| Harbor | Phoenix | | --- | --- | | One task collection | One versioned dataset | | One task | One dataset example | | One distinct agent and model configuration | One experiment | | One planned task attempt | One repetition | | One final logical trial | One experiment run | | Final textual agent turn | Experiment run output | | Final verifier reward | Experiment evaluation with the original key and CODE annotator kind | | Step verifier reward | Evaluation named `<step_name>.<reward_key>` | | Trial or step exception | Run error and `infra_ok=0` | | Saved ATIF trajectories | One trace linked to the run, with one step span per attempted step in a multi-step task |

Each single-step or multi-step Harbor task becomes one Phoenix dataset example. A multi-step example input includes its ordered step names and instructions. Phoenix examples keep `output` empty unless the task declares a reference file.

The plugin records only the terminal physical attempt for a logical trial. An attempt that Harbor will retry does not create a Phoenix run. Completion order does not define repetition numbers.

When a saved terminal ATIF trajectory ends with a user-facing textual agent turn, the plugin records it in chat-message format so Phoenix experiment comparisons render it as Markdown. Structured messages contribute their text parts in order; media parts are omitted. The output stays empty for missing or invalid trajectories, terminal tool calls, media-only turns, and state-only tasks. Multi-step tasks use the last attempted step, and continued trajectories use the terminal continuation. Output extraction still runs with `trace_mode=null`; that setting disables trace creation, not result display.

Successful runs written by older plugin versions keep their legacy Harbor metadata output because Phoenix runs are immutable. Resume recognizes that exact legacy shape and reuses the run.

Add optional reference outputs

A task can declare a checked-in reference file in its root `task.toml`:

[metadata.arize-phoenix]
reference_output_path = "tests/expected.json"

The plugin reads this UTF-8 JSON file during setup. A JSON string becomes `{"messages": [{"role": "assistant", "content": "the reference text"}]}`. A JSON object is stored unchanged as the example output. Other top-level types must be wrapped in an object. No setting means an empty output, even if an `expected.json` file exists.

Paths are relative to the downloaded task root on the Harbor host. Absolute paths, `..` components, and symlinks escaping that root are rejected. Missing, unreadable, or invalid configured files fail setup before trials run. Keep references beside verifier assets, out of the agent workspace.

This works for local datasets, direct tasks, and published tasks. Multi-step tasks have one reference for the whole task, usually its expected final result. Reference-content changes create a dataset version; existing experiments keep their original version. References do not change Harbor's grading.

Name the dataset

The plugin infers a Phoenix dataset name for each supported single-source job. Provide `dataset=<name>` only when a job contains several direct tasks, which have no shared collection name, or when you want to customize the dataset's display name in Phoenix.

The inferred names are:

| Harbor source |