/exploring-llm-clusters

name: exploring-llm-clusters
description: 'Investigate AI observability clusters — understand usage patterns in AI/LLM traffic, compare cluster behavior, compute cost/latency metrics, and drill into individual traces within clusters.'

Exploring LLM clusters

Use this skill when investigating AI observability clusters — understanding what patterns exist in your AI/LLM traffic, comparing cluster behavior, and drilling into individual clusters.

Tools

| Tool | Purpose | | ---------------------------------- | ----------------------------------------------- | | `posthog:llma-clustering-job-list` | List clustering job configurations for the team | | `posthog:llma-clustering-job-get` | Get a specific clustering job by ID | | `posthog:execute-sql` | Query cluster run events and compute metrics | | `posthog:query-llm-traces-list` | Find traces belonging to a cluster | | `posthog:query-llm-trace` | Inspect a specific trace in detail |

How clustering works

PostHog clusters LLM traces, individual generations, or evaluation events by embedding similarity. A Temporal workflow runs periodically or on-demand, producing cluster events stored as `$ai_trace_clusters` (trace-level), `$ai_generation_clusters` (generation-level), or `$ai_evaluation_clusters` (evaluation-level).

Each cluster event contains:

• `$ai_clustering_run_id` — unique run identifier (format: `<team_id>_<level>_<YYYYMMDD>_<HHMMSS>[_<job_id>]`)
• `$ai_clustering_level` — `"trace"`, `"generation"`, or `"evaluation"`
• `$ai_window_start` / `$ai_window_end` — time window of the data that was analyzed
• `$ai_total_items_analyzed` — number of traces, generations, or evaluations processed
• `$ai_clusters` — JSON array of cluster objects
• `$ai_clustering_params` — algorithm parameters used

The analyzed window closes when a run starts, and the cluster event lands once the run finishes. So the cluster event's own `timestamp` is always **after** `$ai_window_end`, by anything from seconds to hours. Use the window only to bound the traces, generations, and evaluations that were analyzed. To find the cluster event itself, filter on `$ai_clustering_run_id` with a plain recent-time bound.

Cluster object shape (inside `$ai_clusters`)

{
  "cluster_id": 0,
  "size": 42,
  "title": "User authentication flows",
  "description": "Traces involving login, signup, and token refresh operations",
  "traces": {
    "<trace_or_generation_id>": {
      "distance_to_centroid": 0.123,
      "rank": 0,
      "x": -2.34,
      "y": 1.56,
      "timestamp": "2026-03-28T10:00:00Z",
      "trace_id": "abc-123",
      "generation_id": "gen-456"
    }
  },
  "centroid_x": -2.1,
  "centroid_y": 1.4
}

• `cluster_id: -1` is the **noise/outlier** cluster (items that didn't fit any cluster)
• Items in `traces` are keyed by trace ID (trace-level), generation event UUID (generation-level), or evaluation event UUID (evaluation-level)
• `rank` orders items by proximity to centroid (0 = closest)
• `x`, `y` are 2D coordinates for visualization (UMAP/PCA/t-SNE reduced)

Clustering jobs

Each team can have up to 10 clustering jobs. A job defines:

• **name** — human-readable label
• **analysis_level** — `"trace"`, `"generation"`, or `"evaluation"`
• **event_filters** — property filters scoping which items are included
• **enabled** — whether the job runs on schedule

Default jobs named `"Default - traces"`, `"Default - generations"`, and `"Default - evaluations"` are auto-created and disabled when a custom job is created for the same level.

Workflow: explore clusters

Step 1 — List recent clustering runs

posthog:execute-sql
SELECT
    toString(properties.$ai_clustering_run_id) AS run_id,
    toString(properties.$ai_clustering_level) AS level,
    toString(properties.$ai_clustering_job_id) AS job_id,
    toString(properties.$ai_clustering_job_name) AS job_name,
    toString(properties.$ai_window_start) AS window_start,
    toString(properties.$ai_window_end) AS window_end,
    toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
    timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
    AND timestamp >= now() - INTERVAL 14 DAY
ORDER BY timestamp DESC
LIMIT 10

Step 2 — Get clusters from a specific run

posthog:execute-sql
SELECT
    toString(properties.$ai_clustering_run_id) AS run_id,
    toString(properties.$ai_clustering_level) AS level,
    toString(properties.$ai_clustering_job_id) AS job_id,
    toString(properties.$ai_clustering_job_name) AS job_name,
    toString(properties.$ai_window_start) AS window_start,
    toString(properties.$ai_window_end) AS window_end,
    toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
    properties.$ai_clusters AS clusters,
    properties.$ai_clustering_params AS params,
    timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
    AND timestamp >= now() - INTERVAL 14 DAY
    AND toString(properties.$ai_clustering_run_id) = '<run_id>'
ORDER BY timestamp DESC
LIMIT 1

Keep the lookback bound wide enough to cover the `timestamp` Step 1 reported for the run. Never bound this query with `$ai_window_start` / `$ai_window_end`. The cluster event is emitted after the window closes, so those bounds return zero rows.

The `clusters` field is a JSON array. Parse it to see cluster titles, sizes, descriptions, optional `metrics`, and each cluster's `traces` map.

**Important:** The clusters JSON can be very large (thousands of trace, generation, or evaluation IDs with coordinates). When the result is too large for inline display, it auto-persists to a file. Use `print_clusters.py` from [scripts/](./scripts/) to get a readable summary.

Step 3 — Compute metrics for cluste