/exploring-llm-traces

name: exploring-llm-traces
description: >
  ABSOLUTE MUST to debug and inspect LLM/AI agent traces using PostHog's MCP tools.
  Use when the user pastes a trace or session URL (e.g. /ai-observability/traces/<id> or /ai-observability/sessions/<id>),
  asks to debug a trace, figure out what went wrong, check if an agent used a tool correctly,
  verify context/files were surfaced, inspect subagent behavior, investigate LLM decisions,
  or analyze token usage and costs. Also use when raw SQL/HogQL against
  `events.properties.$ai_input` / `$ai_output_choices` returns empty — message content lives only
  on the dedicated `posthog.ai_events` table.

Exploring LLM traces with MCP tools

PostHog captures LLM/AI agent activity as traces. Each trace is a tree of events representing a single AI interaction — from the top-level agent invocation down to individual LLM API calls.

Available tools

| Tool | Purpose | | ------------------------------- | ------------------------------------------------------------- | | `posthog:query-llm-traces-list` | Search and list traces; can return large multi-trace payloads | | `posthog:query-llm-trace` | Get a single trace by ID with full event tree | | `posthog:read-data-schema` | Discover custom event/person properties before filtering | | `posthog:execute-sql` | Ad-hoc SQL for complex trace analysis |

Event hierarchy

See the [event reference](./references/events-and-properties.md) for the full schema.

$ai_trace (top-level container)
  └── $ai_span (logical groupings, e.g. "RAG retrieval", "tool execution")
        ├── $ai_generation (individual LLM API call)
        └── $ai_embedding (embedding creation)

Events are linked via `$ai_parent_id` → parent's `$ai_span_id` or `$ai_trace_id`.

Workflow: debug a trace or session from a URL

Step 1 — Classify the URL

First inspect the path. Do not treat every UUID-looking value as a trace ID.

• `/ai-observability/traces/<trace_id>` or legacy `/llm-analytics/traces/<trace_id>` / `/llm-observability/traces/<trace_id>` is a single trace. Fetch it with `posthog:query-llm-trace`.
• `/ai-observability/sessions/<session_id>` or legacy `/llm-analytics/sessions/<session_id>` is an AI session, not a trace. Fetch traces with `posthog:query-llm-traces-list` filtered by event property `$ai_session_id`.

Preserve `date_from` / `date_to` query parameters from the URL when present. If none are present but the URL has a `timestamp` query parameter, use that timestamp as the anchor and query an absolute window around it, for example `timestamp - 36h` to `timestamp + 36h`. This handles exact session links whose UI timestamp may be offset from the stored event timestamps while keeping the query bounded. If the URL has neither explicit dates nor `timestamp`, use a safe default like `{"date_from": "-7d"}`.

For exact trace and session URLs, skip schema discovery for the standard `$ai_*` fields used below. These are AI observability built-ins, not project-specific custom properties.

Step 2 — Fetch trace data

For a trace URL, call `posthog:query-llm-trace` with:

{
  "traceId": "<trace_id>",
  "dateRange": { "date_from": "-7d" }
}

For a session URL, call `posthog:query-llm-traces-list` with:

{
  "dateRange": { "date_from": "<timestamp_minus_36h>", "date_to": "<timestamp_plus_36h>" },
  "filterTestAccounts": false,
  "limit": 20,
  "properties": [{ "type": "event", "key": "$ai_session_id", "value": ["<session_id>"], "operator": "exact" }]
}

Use the URL's `date_from` / `date_to` values in the session query if present. If the URL only has `timestamp`, calculate the absolute date range from that timestamp instead of using a relative range like `-1h`. Set `filterTestAccounts: false` for an exact URL so the requested trace is not hidden by account filters.

The result contains the event tree with all properties. The response may be large — when it exceeds the inline limit, Claude Code auto-persists it to a file.

From the result you get:

• Every event with its type (`$ai_span`, `$ai_generation`, etc.)
• Span names (`$ai_span_name`) — these are the tool/step names
• Latency, error flags, models used
• Parent-child relationships via `$ai_parent_id`
• `_posthogUrl` — **always include this in your response** so the user can click through to the UI

Step 3 — Parse large results with scripts

When the result is persisted to a file (large traces with full `$ai_input`/`$ai_output_choices`), use the [parsing scripts](./scripts/) to explore it.

**Start with the summary** to get the full picture, then drill into specifics:

# 1. Overview: metadata, tool calls, final output, errors
python3 scripts/print_summary.py /path/to/persisted-file.json

# 2. Timeline: chronological event list with truncated I/O
python3 scripts/print_timeline.py /path/to/persisted-file.json

# 3. Drill into a specific span's full input/output
SPAN="tool_name" python3 scripts/extract_span.py /path/to/persisted-file.json

# 4. Full conversation with thinking blocks and tool calls
python3 scripts/extract_conversation.py /path/to/persisted-file.json

# 5. Search for a keyword across all properties
SEARCH="keyword" python3 scripts/search_traces.py /path/to/persisted-file.json

All scripts support `MAX_LEN=N` env var to control truncation (0 = unlimited).

Investigation patterns

"Did the agent use the tool correctly?"

1. Find the `$ai_span` for the tool call (look at `$ai_span_name`) 2. Check `$ai_input_state` — what arguments were passed to the tool? 3. Check `$ai_output_state` — what did the tool return? 4. Check `$ai_is_error` — did the tool call fail?

"Was the context correct?" / "Were the right files surfaced?"

1. Find the `$ai_generation` event where the LLM made the decision 2. Check `$ai_input` — this is the full message history the LLM s