/output-eval-error-analysis

name: output-eval-error-analysis
description: Systematically review workflow traces to identify failure modes before building evaluators. Use when starting an eval project, after significant pipeline changes, or when production quality drops.
allowed-tools: [Bash, Read, Write, Edit]

Error Analysis for Workflow Evaluation

Overview

Review real workflow traces and categorize how your workflow fails **before** writing any evaluators. Evaluators built without error analysis target generic qualities ("is this good?") instead of the specific ways your workflow actually breaks. This skill walks you through the process.

When to Use

• Starting a new eval project for an existing workflow
• Production quality has dropped and you need to understand why
• After significant prompt, model, or pipeline changes
• Before building your first evaluator for a workflow

Step 1: Collect Traces

Gather 50-100 representative workflow executions. More traces = more reliable failure categories.

From recent runs

List recent workflow executions and pull their traces:

# List recent runs for a workflow
npx output workflow runs list <workflowName>

# Pull a specific trace as JSON
npx output workflow debug <workflowId> --json

From production (bulk download)

Download production traces directly into dataset YAML files:

# Download up to 20 recent traces as dataset files
npx output workflow dataset generate <workflowName> --download --limit 20

This creates YAML files in `tests/datasets/` with the `input` and `last_output` fields populated from real executions.

From scenario-driven generation

If production traces are sparse, generate traces from scenario inputs:

# Generate a dataset from a scenario file
npx output workflow dataset generate <workflowName> basic --name basic_trace

# Generate from inline JSON
npx output workflow dataset generate <workflowName> --input '{"topic": "AI safety"}' --name ai_safety_trace

Run enough inputs to get 50+ traces. Prioritize diversity over volume — vary inputs across the dimensions you expect to matter.

Step 2: Review Traces Individually

Review each trace one at a time. For each trace, record:

| Field | What to write | |-------|---------------| | **Trace ID** | The workflow execution ID | | **Verdict** | Pass or Fail (binary — no "partial" at this stage) | | **Root cause** | If Fail: what specifically went wrong and why | | **Notes** | Anything surprising or worth remembering |

Review template

Create a file to track your reviews. A simple markdown table works:

# Error Analysis: <workflow_name>
# Date: YYYY-MM-DD
# Traces reviewed: 0 / 50

| # | Trace ID | Verdict | Root Cause | Notes |
|---|----------|---------|------------|-------|
| 1 | abc-123  | Fail    | Hallucinated a URL that doesn't exist | Common with technical topics |
| 2 | def-456  | Pass    | — | Clean output |
| 3 | ghi-789  | Fail    | Ignored the "formal tone" requirement | Input had conflicting signals |

What to look for in each trace

Open the JSON trace and examine:

1. **Final output** — Does it meet the user's intent? Is it correct? 2. **Step-by-step data flow** — Did each step receive the right input and produce reasonable output? 3. **LLM responses** — Did the model follow instructions? Did it hallucinate? 4. **Error states** — Did any step fail, retry, or produce unexpected errors?

Critical rule: read first, categorize second

Review at least 30 traces before naming any failure categories. Premature categorization causes you to see patterns that aren't there and miss patterns that are. Just record what you observe.

Step 3: Group Into Failure Categories

After reviewing 30+ traces, patterns will emerge. Group your failures into 5-10 categories based on **root cause**, not surface symptoms.

Good categories (root cause)

• "Hallucinated URLs" — model invents links that don't exist
• "Tone mismatch" — output tone doesn't match the requested persona
• "Missing required section" — output omits a section the input explicitly requested
• "Factual error" — output contains verifiably wrong claims
• "Prompt injection leak" — user input manipulates the system prompt

Bad categories (surface symptoms)

• "Bad output" — too vague, not actionable
• "LLM error" — doesn't identify the specific failure
• "Quality issue" — could mean anything

Splitting and merging

• If a category has fewer than 3 examples, merge it into a broader category or note it as rare
• If a category has 15+ examples and contains distinct sub-patterns, split it
• Categories should be **mutually exclusive** — each failure belongs to exactly one category

Example categorization

For a blog generation workflow after reviewing 60 traces:

| Category | Count | Rate | Example | |----------|-------|------|---------| | Hallucinated URLs | 8 | 13% | Invented links to non-existent pages | | Tone mismatch | 6 | 10% | Casual tone when formal was requested | | Off-topic drift | 5 | 8% | Blog about "AI" drifted to unrelated ML history | | Missing sections | 4 | 7% | Skipped "conclusion" when explicitly requested | | Too short | 3 | 5% | Under 200 words when 500+ requested | | **Total failures** | **26** | **43%** | | | **Passes** | **34** | **57%** | |

Step 4: Label Datasets

Add `ground_truth` labels to your dataset YAML files so evaluators can validate against them. Each failure category maps to a future evaluator name.

YAML structure

name: ai_safety_trace
input:
  topic: "AI safety"
  tone: "formal"
  min_length: 500
last_output:
  output:
    title: "Understanding AI Safety"
    blog_post: "AI safety is super important and stuff..."
  executionTimeMs: 3200
  date: '2026-03-25T00:00:00.000Z'
ground_truth:
  # Global ground truth (available to all evaluators)
  human_verdict: fail
  failure_categories:
    - tone_mismatch
  notes: "Used casual language despite formal tone request"
  # Per-evaluator ground truth
  e