/exploring-apm-traces

name: exploring-apm-traces
description: >
  Investigates distributed application performance using PostHog APM (OpenTelemetry span) data via MCP.
  Use when the user asks about service traces, slow HTTP/database spans, error spans, error-rate trends or
  spikes, latency distributions, trace IDs, or span attributes — not AI observability traces or product logs.
  Uses posthog:apm-service-health, posthog:query-apm-spans, posthog:apm-trace-get, posthog:apm-spans-sparkline,
  posthog:apm-services-list, posthog:apm-attributes-list, and posthog:apm-attribute-values-list.

Exploring APM traces (OpenTelemetry spans)

PostHog captures distributed traces from OpenTelemetry. Each trace is a tree of spans representing a request's path through services.

**Disambiguation:** This skill is for **APM / OpenTelemetry traces**. Do not confuse with **AI observability traces** (agent/model `$ai_*` events) or **logs** (`posthog:query-logs`, `posthog:logs-*`).

Governed metric first

When the question asks for SLO burn, call `posthog:metric-list` before the APM tools and look for `slo_explicit_burn_by_operation`. Run an approved, non-drifted match with `posthog:data-catalog-metric-run` for the canonical headline. If the user also asks which operation drives the burn, answer the headline first, then use the aggregate and trace workflows below for a noncanonical breakdown. If no governed metric matches, say so and label the derived measure noncanonical.

Available tools

| Tool | Purpose | | -------------------------------------- | ------------------------------------------------- | | `posthog:apm-service-health` | Status per service against a baseline: start here | | `posthog:query-apm-spans` | Search and filter spans (compact list view) | | `posthog:apm-trace-get` | Summarize one hex `trace_id` (or list its spans) | | `posthog:apm-spans-aggregate` | Per-operation aggregates (count, p50/p95, errors) | | `posthog:apm-spans-tree` | Call-tree aggregates per `(parent, child)` edge | | `posthog:apm-spans-count` | Scalar span count — cheap filter pre-flight | | `posthog:apm-spans-sparkline` | Span counts over time (zero-filled time series) | | `posthog:apm-spans-duration-histogram` | Trace counts per log-scale duration bucket | | `posthog:apm-attribute-breakdown` | Span counts grouped by one attribute's value | | `posthog:apm-services-list` | List distinct service names | | `posthog:apm-attributes-list` | List span or resource attribute keys | | `posthog:apm-attribute-values-list` | List values for a specific attribute key |

See [references/spans-and-fields.md](./references/spans-and-fields.md) for the response schema and the `kind`/`status_code` enums.

Workflow: debug a trace from a URL

Step 1 — Fetch the trace

posthog:apm-trace-get
{
  "trace_id": "<hex_trace_id>"
}

By default the response is `{ summary: {…}, results: [], _posthogUrl: "…" }`: a digest of about 3 KB, whatever the trace size. Entries name their `service` only when it differs from the root's.

• `duration_ms`, `span_count`, `service_count`, `error_count`, `root`, and `services` (spans per service)
• `top_self_time` — spans with the most self time (duration not covered by children); a parent near the top is an **uninstrumented gap**
• `critical_path` — the spans the trace's end time waits on, from the root down (`depth`), with a run of same-named siblings collapsed into one entry with a `count`; spans under 1% of the root's duration are left out and counted in `critical_path_omitted`
• `repeated_spans` — service and name pairs that occur more than once, by total time; many fast repeats are an N+1 pattern
• `errors` — error spans with the exception `message`
• `truncated` — true when the trace has more than 2000 spans and the digest covers only the first 2000
• The `_posthogUrl` — a deep link to this trace in the tracing UI; **always include this in your response** so the user can click through

Use the `span_id`s in the summary to say which span you mean, and pass `summaryLimit` (up to 50) for longer lists.

Step 2 — Fetch every span only when the summary is not enough

Pass `"summary": false` to get `{ results: [span, span, …], hasMore, nextOffset }`: every span with `name`, `service_name`, `kind`, `status_code`, `parent_span_id`, `duration_nano`, `self_time_nano`, `is_root_span`, its `attributes` and `events`. Set `"excludeAttributes": true` unless you need the attributes. The list can be very large for fan-out request flows; when it exceeds the inline limit, Claude Code auto-persists it to a file.

When the result is persisted to a file (traces with hundreds of spans across services), use the [parsing scripts](./scripts/) to explore it.

**Start with the summary** to get the full picture, then drill into specifics:

# 1. Overview: services, span count, slowest spans, errors
python3 scripts/print_summary.py /path/to/persisted-file.json

# 2. Indented chronological tree (DFS by parent_span_id)
python3 scripts/print_timeline.py /path/to/persisted-file.json

# 3. Drill into a specific span by name
SPAN="HTTP GET /api/users" python3 scripts/extract_span.py /path/to/persisted-file.json

# 4. Search for a keyword across span names, services, IDs
SEARCH="keyword" python3 scripts/search_spans.py /path/to/persisted-file.json

# 5. When the JSON shape looks unfamiliar
python3 scripts/show_structure.py /path/to/persisted-file.json

All scripts support `MAX_LEN=N` env var to control truncation (`0` = unlimited).

Tree reconstruction (parent_span_id → span_id)

The flat span list is a tree. Each span carries:

• `trace_id` — same on every span in the trace
• `span_id` — this span's unique hex ID
• `parent_span_id` — points to the parent's `span_id`