/phoenix-error-analysis

name: phoenix-error-analysis
description: Find out what is going wrong in LLM or agent traffic by reading sampled Phoenix traces, spans, or sessions, writing free-form notes (open coding), then grouping the notes into a few narrow annotations, one per failure dimension with a small label set and counts, that pick eval targets and fix priorities (axial coding). Use for "what's going wrong with this agent", "I just instrumented my app, where do I start", "review these traces", "the chatbot keeps losing context", "what kinds of mistakes is the model making", "what categories of failures do we have", "what should I build evals for", "how do I prioritize fixes", "group these notes", "MECE breakdown" — or any framing that needs observations or categories grounded in real traces rather than invented top-down, even without naming the technique.
summary: Read sampled traces, write open-coding notes, then group them into narrow one-dimension annotations with counts that pick eval targets and fix priorities.
license: Apache-2.0
metadata:
  author: arize-ai
  version: "1.0.0"

Error Analysis

Two phases against sampled traces, spans, or sessions. **Open coding** reads each sampled entity and writes a short, specific note on what went wrong. **Axial coding** groups those notes into a few narrow annotations, each judging one dimension of the entity with a small label set, and counts the labels to feed eval design and fix prioritization. Open coding always comes first: dimensions and labels that are not grounded in notes are invented top-down, which is the failure this workflow exists to avoid. Axial coding can also start from any existing set of open-ended observations.

**Recorded notes are the deliverable.** Open coding ends with one note per problematic entity written to the server and mirrored in the local sidecar, not with a summary in chat. Reading traces and reporting what you noticed without recording anything is a read-only diagnosis, the top-down shortcut this workflow exists to replace. A project-wide question such as "find any issues", "what's going wrong", or "any errors here" is a request for phase 1: pick the identifier, pick the unit, record a note for each problem as you find it, then summarize. A project with no annotation names yet is the normal starting state, not a reason to hold off writing.

Choosing the unit of analysis

Pick **trace, span, or session** deliberately before recording — the choice determines what every note and annotation below targets, and a wrong default is expensive to undo mid-run. The unit is where the failure modes you're investigating actually live:

• **Trace** — one input → one output. Classifiers, single-shot summarizers, stateless tool-using agents, single-query RAG.
• **Span** — one operation inside a trace. In-isolation mechanical failures (exception, tool error response, malformed output) or failures attributable on sight to one component.
• **Session** — a sequence of traces sharing a `session.id`. Multi-turn agents where the failure is a *trajectory* — context loss across turns, goal drift, forgotten preferences — that exists only *across* traces.

Three signals to read:

1. **User framing.** "Conversation", "agent forgot", "drift", "across turns" → session. "This trace", "wrong output" → trace. "Exception", "malformed", "the retrieval failed" → span.

2. **Session wiring.** The session id lives at the root span's `session.id` attribute (not a top-level trace field); `""` means absent. List ~200 recent traces and compute how many carry a non-empty session id, how many distinct session ids appear, and the median traces per session. No session ids or median 1 → trace or span level; median 2+ → session level is plausibly right.

3. **Turn structure.** Open one recent trace's root-span `input.value` (it may be large — filter rather than reading wholesale). A single user message → one shot; a message *array* (`[{role: user}, {role: assistant}, ...]`) → a turn of a dialogue that lives at the session level.

State the chosen unit explicitly before recording any note. It can shift if the data demands it — trace-level notes that keep surfacing "the agent never remembers earlier turns" should pivot the next batch to session. The unit is a starting hypothesis, not a contract.

Coding annotation identifier (pick this first)

Every artifact this workflow produces — open-coding notes, axial-coding labels, and the local sidecar files — is tagged with one **coding annotation identifier** so the run is queryable and revertible as a unit. Pick a descriptive, unique value before recording, e.g. `coding-run:chatbot-context-loss-2026-05-06`. (The `coding-run:` prefix is a naming convention; the value is not a Phoenix session id.)

Pass the identifier explicitly on every server write. A shell variable is fine for readability, but do not rely on shell inheritance — many agent harnesses spawn each command in a fresh subshell.

The local sidecars live at `.px/coding/<sanitized-identifier>.jsonl` (open-coding notes) and `.px/coding/<sanitized-identifier>-axial.jsonl` (axial labels), CWD-relative; sanitization replaces every character outside `[a-zA-Z0-9_-]` with `-`. Uniqueness is a **local file check**, not a server query:

CODING_ANNOTATION_IDENTIFIER="coding-run:chatbot-context-loss-2026-05-06"
SLUG=$(echo -n "$CODING_ANNOTATION_IDENTIFIER" | sed 's/[^a-zA-Z0-9_-]/-/g')
NOTES_SIDECAR=".px/coding/${SLUG}.jsonl"
AXIAL_SIDECAR=".px/coding/${SLUG}-axial.jsonl"
test ! -f "$NOTES_SIDECAR" || { echo "Sidecar already exists at $NOTES_SIDECAR — pick a new identifier or delete the file"; exit 1; }
mkdir -p .px/coding

If `$NOTES_SIDECAR` already exists, append a disambiguator (`-v2`, `-dustin`, etc.) and re-check.

**Resuming a run** — the two phases may run in independent invocations. When axial coding starts in a fresh shell, set the identifier to the same value chosen during open coding (recoverable from the wrap-up UI URL or by listing `.px/coding/