/exploring-apm-traces

name: exploring-apm-traces
description: >
  Investigates distributed application performance using PostHog APM (OpenTelemetry span) data via MCP.
  Use when the user asks about service traces, slow HTTP/database spans, error spans, error-rate trends or
  spikes, latency distributions, trace IDs, or span attributes — not AI observability traces or product logs.
  Uses posthog:query-apm-spans, posthog:apm-trace-get, posthog:apm-spans-sparkline,
  posthog:apm-services-list, posthog:apm-attributes-list, and posthog:apm-attribute-values-list.

Exploring APM traces (OpenTelemetry spans)

PostHog captures distributed traces from OpenTelemetry. Each trace is a tree of spans representing a request's path through services.

**Disambiguation:** This skill is for **APM / OpenTelemetry traces**. Do not confuse with **AI observability traces** (agent/model `$ai_*` events) or **logs** (`posthog:query-logs`, `posthog:logs-*`).

Governed metric first

When the question asks for SLO burn, call `posthog:metric-list` before the APM tools and look for `slo_explicit_burn_by_operation`. Run an approved, non-drifted match with `posthog:data-catalog-metric-run` for the canonical headline. If the user also asks which operation drives the burn, answer the headline first, then use the aggregate and trace workflows below for a noncanonical breakdown. If no governed metric matches, say so and label the derived measure noncanonical.

Available tools

| Tool | Purpose | | -------------------------------------- | ------------------------------------------------- | | `posthog:query-apm-spans` | Search and filter spans (compact list view) | | `posthog:apm-trace-get` | Get the full span list for one hex `trace_id` | | `posthog:apm-spans-aggregate` | Per-operation aggregates (count, p50/p95, errors) | | `posthog:apm-spans-tree` | Call-tree aggregates per `(parent, child)` edge | | `posthog:apm-spans-count` | Scalar span count — cheap filter pre-flight | | `posthog:apm-spans-sparkline` | Span counts over time (zero-filled time series) | | `posthog:apm-spans-duration-histogram` | Trace counts per log-scale duration bucket | | `posthog:apm-attribute-breakdown` | Span counts grouped by one attribute's value | | `posthog:apm-services-list` | List distinct service names | | `posthog:apm-attributes-list` | List span or resource attribute keys | | `posthog:apm-attribute-values-list` | List values for a specific attribute key |

See [references/spans-and-fields.md](./references/spans-and-fields.md) for the response schema and the `kind`/`status_code` enums.

Workflow: debug a trace from a URL

Step 1 — Fetch the trace

posthog:apm-trace-get
{
  "trace_id": "<hex_trace_id>"
}

The response is `{ results: [span, span, …], _posthogUrl: "…" }` — a flat list of every span in the trace. The list can be very large for fan-out request flows; when it exceeds the inline limit, Claude Code auto-persists it to a file.

From the result you get:

• Every span with `name`, `service_name`, `kind`, `status_code`, `parent_span_id`, `duration_nano`, `is_root_span`
• The `_posthogUrl` — a deep link to this trace in the tracing UI; **always include this in your response** so the user can click through

Step 2 — Parse large results with scripts

When the result is persisted to a file (traces with hundreds of spans across services), use the [parsing scripts](./scripts/) to explore it.

**Start with the summary** to get the full picture, then drill into specifics:

# 1. Overview: services, span count, slowest spans, errors
python3 scripts/print_summary.py /path/to/persisted-file.json

# 2. Indented chronological tree (DFS by parent_span_id)
python3 scripts/print_timeline.py /path/to/persisted-file.json

# 3. Drill into a specific span by name
SPAN="HTTP GET /api/users" python3 scripts/extract_span.py /path/to/persisted-file.json

# 4. Search for a keyword across span names, services, IDs
SEARCH="keyword" python3 scripts/search_spans.py /path/to/persisted-file.json

# 5. When the JSON shape looks unfamiliar
python3 scripts/show_structure.py /path/to/persisted-file.json

All scripts support `MAX_LEN=N` env var to control truncation (`0` = unlimited).

Tree reconstruction (parent_span_id → span_id)

The flat span list is a tree. Each span carries:

• `trace_id` — same on every span in the trace
• `span_id` — this span's unique hex ID
• `parent_span_id` — points to the parent's `span_id` (zero-padded hex `000…000` for the root)
• `is_root_span` — convenience flag for the trace entry

To rebuild the tree:

1. Spans where `is_root_span` is true (or `parent_span_id == "00000000…"`) are **root spans**. 2. Every other span is a child of the span whose `span_id` matches its `parent_span_id`. 3. Group by `parent_span_id`, walk from each root downward.

`scripts/print_timeline.py` does this for you and prints a DFS-indented tree.

Investigation patterns

"Where is time going?"

1. Every span from `apm-trace-get` carries `self_time_nano` — duration not covered by children. Sort by it: the top span is where wall-clock actually went. A parent with large `self_time_nano` is an **uninstrumented gap** (the work happened inside it, not in any recorded child). 2. Run `print_summary.py` — it surfaces the top-5 slowest spans by `duration_nano`. 3. For a noisy trace, run `print_timeline.py` and scan the indented durations — you can see whether time is dominated by one child span or fan-out across many. 4. To dig into one slow span, `SPAN="<name>" python3 scripts/extract_span.py FILE`. 5. For aggregate "which child dominates" questions use `apm-spans-tree` and read `calls_per_parent_invocation` — it separates a child that's slow per call from one that merely runs 20× per