/error-analysis

name: error-analysis
license: MIT
compatibility: "Claude Code 2.1.277+. Needs Langfuse credentials in env or an exported traces JSONL."
description: "Evals-first error analysis for LLM apps: clusters real Langfuse or JSONL traces into a human-confirmed failure taxonomy with counts, then recommends binary pass/fail evals for recurring named modes. Use to learn what to measure before writing evals. Not for CI failures."
context: fork
# user-typed commands stay interactive; CC >= 2.1.218 backgrounds forks by default (#3093)
background: false
version: 1.0.0
author: OrchestKit
tags: [error-analysis, evaluation, llm-testing, testing, langfuse, traces, failure-taxonomy, evals, debugging]
user-invocable: false
disable-model-invocation: false
allowed-tools: [AskUserQuestion, Bash, Read, Write, Edit, Grep, Glob, Agent, TaskCreate, TaskUpdate, TaskList, TaskGet, TaskStop, WebFetch, WebSearch]
skills: [testing-llm, memory]
complexity: high
persuasion-type: discipline
effort: high
model: sonnet
metadata:
  category: workflow-automation
triggers:
  keywords: ["error analysis", "failure modes", "failure taxonomy", "open coding", "axial coding", "analyze traces", "langfuse traces", "evals first", "what to measure", "why does my agent fail"]
  examples:
    - "run error analysis on my Langfuse traces"
    - "my agent keeps failing and I do not know which evals to write"
    - "pull 100 production traces and build me a failure taxonomy"
  anti-triggers: [ci-debug, errors, "write evals", verify, cover, "fix this bug"]

Error Analysis

Evals-first failure analysis for LLM applications. The method (Hamel Husain, Shreya Shankar) is qualitative research applied to traces: a human open-codes real failures, Claude axial-codes the notes into a named taxonomy, and only the recurring named modes earn an automated eval. Error analysis decides what to measure; it never writes an eval for a mode that has no name and no count.

When to Use

• An LLM feature (chatbot, RAG, agent, extractor) misbehaves and the team is guessing which evals to write
• You have production traces in Langfuse (or a JSONL export) and need the top failure modes with counts
• Eval scores exist but nobody trusts them because they were never grounded in real failures
• A judge or metric exists and its agreement with human labels is unknown

Do NOT use for Claude Code session errors (`errors` skill), failing CI runs (`ci-debug`), or writing the evaluators themselves once modes are named (`testing-llm`, `ork:eval-runner`).

Task Management (CC 2.1.16)

Multi-phase workflow: create tasks before Phase 1 and keep status current.

t_run  = TaskCreate(subject="Error analysis: {target}", activeForm="Running error analysis on {target}")
t_pull = TaskCreate(subject="Pull traces", activeForm="Pulling traces")
t_open = TaskCreate(subject="Open-code failure notes", activeForm="Open-coding traces")
t_axial = TaskCreate(subject="Axial-code failure taxonomy", activeForm="Axial-coding notes")
t_tax  = TaskCreate(subject="Write failure-taxonomy.md", activeForm="Writing taxonomy")
t_eval = TaskCreate(subject="Recommend evals + judge alignment", activeForm="Recommending evals")
TaskUpdate(taskId=t_open,  addBlockedBy=[t_pull])
TaskUpdate(taskId=t_axial, addBlockedBy=[t_open])
TaskUpdate(taskId=t_tax,   addBlockedBy=[t_axial])
TaskUpdate(taskId=t_eval,  addBlockedBy=[t_tax])

Effort Scaling (CC 2.1.76)

| Effort | Trace pool | Notes | Outcome | |--------|-----------|-------|---------| | low | 30 | human notes only, no Claude drafts | draft taxonomy | | medium | 50 | drafts after 10 human notes | taxonomy + counts | | high (default) | 100 | drafts after 30 human notes, saturation check | full deliverable |

Phase 1: Pull Traces

Goal: a working pool of ~100 diverse traces (default N=100), skewed toward failures.

**Source A, Langfuse public API.** Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST` in env. Never echo or print the secret value; reference the variable names only.

mkdir -p error-analysis/traces
# Credentials ride in the Authorization header, so let curl enforce the scheme:
# --proto '=https' fails closed on every non-HTTPS URL. Do not pattern-match the
# host yourself; uppercase schemes, scheme-less names, decimal IPs, and userinfo
# tricks all bypass a regex. Relax to '=http,https' ONLY for an exact local
# endpoint (http://localhost, http://127.0.0.1, http://[::1], optional port and
# path, no userinfo). For a self-hosted internal instance, use HTTPS or an SSH
# tunnel to localhost.
proto="=https"
host_lc=$(printf '%s' "$LANGFUSE_HOST" | tr 'A-Z' 'a-z')
case "$host_lc" in
  http://*)
    authority=${host_lc#http://}
    authority=${authority%%/*}
    case "$authority" in
      *@*) printf '%s\n' 'Refusing HTTP with userinfo in the URL; use HTTPS.' >&2; exit 1 ;;
    esac
    case "$authority" in
      localhost|localhost:*|127.0.0.1|127.0.0.1:*|\[::1\]|\[::1\]:*) proto="=http,https" ;;
      *) printf '%s\n' 'Refusing plain HTTP to a non-local host; use HTTPS or an SSH tunnel to localhost.' >&2; exit 1 ;;
    esac
    ;;
esac
# Langfuse v4 removed GET /api/public/traces (404); the v2 Observations API is the
# supported read. It returns observation rows, so group by traceId client-side.
# io is required in fields or the rows come back without input/output.
curl -sS --fail-with-body --proto "${proto:-=https}" -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
  "$LANGFUSE_HOST/api/public/v2/observations?fromStartTime=$FROM&toStartTime=$TO&limit=100&fields=core,basic,trace_context,io" \
  -o error-analysis/traces/page1.json

Paginate with the `cursor` from each response until ~100 distinct `traceId` values are collected (or the cursor is exhausted), aborting on any non-2xx status so an error body is not read as the end of the cursor. Then finish the selected traces: the last trace in the pool may be missing rows, so keep paging until every selected `traceId` has its root observation (the `is