/exploring-llm-clusters

name: exploring-llm-clusters
description: 'Investigate AI observability clusters — understand usage patterns in AI/LLM traffic, compare cluster behavior, compute cost/latency metrics, and drill into individual traces within clusters.'

Exploring LLM clusters

Use this skill when investigating AI observability clusters — understanding what patterns exist in your AI/LLM traffic, comparing cluster behavior, and drilling into individual clusters.

Tools

| Tool | Purpose | | ---------------------------------- | ----------------------------------------------- | | `posthog:llma-clustering-job-list` | List clustering job configurations for the team | | `posthog:llma-clustering-job-get` | Get a specific clustering job by ID | | `posthog:execute-sql` | Query cluster run events and compute metrics | | `posthog:query-llm-traces-list` | Find traces belonging to a cluster | | `posthog:query-llm-trace` | Inspect a specific trace in detail |

How clustering works

PostHog clusters LLM traces, individual generations, or evaluation events by embedding similarity. A Temporal workflow runs periodically or on-demand, producing cluster events stored as `$ai_trace_clusters` (trace-level), `$ai_generation_clusters` (generation-level), or `$ai_evaluation_clusters` (evaluation-level).

Each cluster event contains:

• `$ai_clustering_run_id` — unique run identifier (format: `<team_id>_<level>_<YYYYMMDD>_<HHMMSS>[_<job_id>]`)
• `$ai_clustering_level` — `"trace"`, `"generation"`, or `"evaluation"`
• `$ai_window_start` / `$ai_window_end` — time window analyzed
• `$ai_total_items_analyzed` — number of traces, generations, or evaluations processed
• `$ai_clusters` — JSON array of cluster objects
• `$ai_clustering_params` — algorithm parameters used

Cluster object shape (inside `$ai_clusters`)

{
  "cluster_id": 0,
  "size": 42,
  "title": "User authentication flows",
  "description": "Traces involving login, signup, and token refresh operations",
  "traces": {
    "<trace_or_generation_id>": {
      "distance_to_centroid": 0.123,
      "rank": 0,
      "x": -2.34,
      "y": 1.56,
      "timestamp": "2026-03-28T10:00:00Z",
      "trace_id": "abc-123",
      "generation_id": "gen-456"
    }
  },
  "centroid_x": -2.1,
  "centroid_y": 1.4
}

• `cluster_id: -1` is the **noise/outlier** cluster (items that didn't fit any cluster)
• Items in `traces` are keyed by trace ID (trace-level), generation event UUID (generation-level), or evaluation event UUID (evaluation-level)
• `rank` orders items by proximity to centroid (0 = closest)
• `x`, `y` are 2D coordinates for visualization (UMAP/PCA/t-SNE reduced)

Clustering jobs

Each team can have up to 10 clustering jobs. A job defines:

• **name** — human-readable label
• **analysis_level** — `"trace"`, `"generation"`, or `"evaluation"`
• **event_filters** — property filters scoping which items are included
• **enabled** — whether the job runs on schedule

Default jobs named `"Default - traces"`, `"Default - generations"`, and `"Default - evaluations"` are auto-created and disabled when a custom job is created for the same level.

Workflow: explore clusters

Step 1 — List recent clustering runs

posthog:execute-sql
SELECT
    toString(properties.$ai_clustering_run_id) AS run_id,
    toString(properties.$ai_clustering_level) AS level,
    toString(properties.$ai_clustering_job_id) AS job_id,
    toString(properties.$ai_clustering_job_name) AS job_name,
    toString(properties.$ai_window_start) AS window_start,
    toString(properties.$ai_window_end) AS window_end,
    toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
    timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
    AND timestamp >= now() - INTERVAL 14 DAY
ORDER BY timestamp DESC
LIMIT 10

Step 2 — Get clusters from a specific run

posthog:execute-sql
SELECT
    toString(properties.$ai_clustering_run_id) AS run_id,
    toString(properties.$ai_clustering_level) AS level,
    toString(properties.$ai_clustering_job_id) AS job_id,
    toString(properties.$ai_clustering_job_name) AS job_name,
    toString(properties.$ai_window_start) AS window_start,
    toString(properties.$ai_window_end) AS window_end,
    toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
    properties.$ai_clusters AS clusters,
    properties.$ai_clustering_params AS params,
    timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND toString(properties.$ai_clustering_run_id) = '<run_id>'
ORDER BY timestamp DESC
LIMIT 1

The `clusters` field is a JSON array. Parse it to see cluster titles, sizes, descriptions, optional `metrics`, and each cluster's `traces` map.

**Important:** The clusters JSON can be very large (thousands of trace, generation, or evaluation IDs with coordinates). When the result is too large for inline display, it auto-persists to a file. Use `print_clusters.py` from [scripts/](./scripts/) to get a readable summary.

Step 3 — Compute metrics for clusters

For trace-level clusters, compute cost/latency/token metrics:

posthog:execute-sql
SELECT
    properties.$ai_trace_id as trace_id,
    sum(toFloat(properties.$ai_total_cost_usd)) as total_cost,
    max(toFloat(properties.$ai_latency)) as latency,
    sum(toInt(properties.$ai_input_tokens)) as input_tokens,
    sum(toInt(properties.$ai_output_tokens)) as output_tokens,
    countIf(properties.$ai_is_error = 'true') as error_count
FROM events
WHERE event IN ('$ai_generation', '$ai_embedding', '$ai_span')
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND