/exploring-ai-failures

name: exploring-ai-failures
description: >
  Find where an AI/LLM application is failing in production and surface the failure patterns, working from
  real traces. Use when someone wants to understand what's going wrong with an AI feature, find and
  categorize failure modes, triage errors, or investigate quality issues (wrong answers, ignored
  instructions, hallucinations, tool misuse) — "what's failing in my agent", "surface error patterns",
  "why are the responses bad", "find the common failure modes", "what should I fix next". Covers scoping
  to one use case, finding failing traces by whichever signal fits the context (code errors, metric
  outliers, trace-type slices, manual review, existing-eval spikes, clustering), and reading them into a
  ranked failure taxonomy.

Exploring AI failures

The highest-value thing you can do with production AI traffic is look at where it fails and name the patterns. The catch: **most failures are silent.** The model returns a clean response — HTTP 200, no exception — that is wrong, off-topic, ignores an instruction, or misuses a tool. Those never raise an error, and they're usually the failures worth caring about.

So this skill is about finding failures (loud _and_ silent), **reading them**, and grouping them into a **ranked set of failure modes** you can act on: fix a prompt, file a bug, prioritize work, or turn the top mode into an automatic eval (`creating-online-evaluations`).

**Everything below serves one irreducible activity: reading real traces.** The queries only tell you _which_ traces to open — they are never the answer. If you report a list of problems without having opened traces, you've described the loud minority (the things that throw errors) and missed the job.

This is bottom-up: the failure modes emerge from real traces, not from a list of generic metrics decided in advance. For reading a single trace in depth, lean on `exploring-llm-traces`; for emergent grouping at high volume, `exploring-llm-clusters`.

Tools

| Tool | Purpose | | ------------------------------- | ------------------------------------------------------------------------ | | `posthog:query-llm-traces-list` | List candidate traces — filter by error, sort by a metric, scope by type | | `posthog:query-llm-trace` | Read a trace in full to see what actually went wrong | | `posthog:execute-sql` | Find metric outliers, discover the trace taxonomy, count failure modes | | `posthog:llma-evaluation-list` | Find existing evals whose failures might reveal a new mode | | `posthog:generate-app-url` | Build a region- and project-qualified deep link to a trace or list |

Detailed queries for each strategy below are in [references/finding-traces.md](references/finding-traces.md). The full `$ai_*` event schema (and the `events` vs `ai_events` split for heavy content like `$ai_input`/`$ai_output_choices`) lives in `exploring-llm-traces/references/events-and-properties.md`.

Work with the user

Collaborate on _scope and priorities_ — not on whether to do the work. Narrow with the user up front: which feature or use case? have they already seen something bad? is there a signal to follow (a thumbs-down, a ticket, a metric that looks off)? Once it's scoped, **go read traces and come back with coded failure modes** — don't stop to ask permission before the reading; that reading is the core activity, not an optional follow-up to offer. When the user doesn't know what to look for, drive the loop below and explain the reasoning as you go; keep the teaching opt-in.

Step 1 — Scope to one use case

Apps have a _taxonomy_ of trace types, and each fails differently — a support chat hallucinates policy, a summarizer drops key points, an agent loops or misuses a tool. Evaluating or analyzing them together averages the signal away. **Pick one**, then find its filter (a `$ai_trace_id` prefix, a feature property, a model). If the user isn't sure how their traffic splits, discover the taxonomy first (query in [references/finding-traces.md](references/finding-traces.md)).

Step 2 — Pick which traces to read

These are ways to _select which traces to open_ — not answers in themselves. The queryable ones (error counts, metric aggregates) tell you _where to look_; they are never the output. Choose by the context and signals you have, and combine them:

• **Code errors (`$ai_is_error`)** — the cheapest sweep and the _least_ representative signal: it only

catches exceptions and API failures, not the silent quality failures that matter most. Use it to grab a few traces to read, not as a tally of "the problems." Slightly more useful for structured-output or tool-calling pipelines, where some failures do surface as parse/schema errors.

• **Metric outliers** — sort by output/input tokens, message length, cost, or latency and open the

extremes. Runaway length, truncation, context bloat, and loops cluster at the tails.

• **One trace-type slice** — narrow to a single kind of request so the traces you read share a taxonomy.
• **Stratified sample** — when you have no specific signal (the common case), pull a mixed batch across

slices and outcomes and read it. This is the default, not the fallback.

• **Existing-eval spikes** — when evals already run, a jump in an eval's failures points you at traces to

read (`llma-evaluation-list` + `execute-sql` over the `$ai_evaluation` events).

• **Clustering** — at high volume, let groupings emerge to pick representative traces to read; see

`exploring-llm-clusters`.

> **The trap.** It's tempting to `GROUP BY` error messages, produce a ranked table, and stop. That table > is the loud minority — failures that raise an exception. The failures that matter for most AI products > complete with HTTP 200 and only appear when a human reads the trace. **A ranking built from error or > metric c