/exploring-llm-evaluations

name: exploring-llm-evaluations
description: >
  Investigate AI observability evaluations — `hog` (deterministic code-based),
  `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment).
  Find existing evaluations, inspect their configuration, run them against
  specific generations, query individual results, and generate AI-powered
  summaries for boolean pass/fail runs. Use when the user asks to debug why an
  evaluation is failing, surface common failure modes, compare results across
  filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, inspect
  sentiment classifications, or manage the evaluation lifecycle.

Exploring AI observability evaluations

PostHog evaluations score `$ai_generation` events. Each evaluation is one of three types:

• **`hog`** — deterministic Hog code that returns `true`/`false` (and optionally N/A).

Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code.

• **`llm_judge`** — an LLM scores generations against a prompt you write. Best for

subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic drift, instruction-following. Costs an LLM call per run and requires AI data processing approval at the org level.

• **`sentiment`** — classifies sentiment from user messages on each matching

generation. Returns a sentiment label and score, not a pass/fail verdict.

Results from all types land in ClickHouse as `$ai_evaluation` events. Boolean evaluations (`llm_judge` and `hog`) set `$ai_evaluation_result`; sentiment evaluations set `$ai_sentiment_*` properties instead.

This skill covers the full lifecycle: list/inspect/manage evaluation configs, run them on specific generations, query individual results, and get an AI-generated summary of pass/fail/N/A patterns across many boolean runs.

Tools

| Tool | Purpose | | ---------------------------------------- | -------------------------------------------------------------- | | `posthog:llma-evaluation-list` | List/search evaluation configs (filter by name, enabled flag) | | `posthog:llma-evaluation-get` | Get a single evaluation config by UUID | | `posthog:llma-evaluation-create` | Create a new `llm_judge`, `hog`, or `sentiment` evaluation | | `posthog:llma-evaluation-update` | Update an existing evaluation (name, prompt, enabled, …) | | `posthog:llma-evaluation-delete` | Soft-delete an evaluation | | `posthog:llma-evaluation-run` | Run an evaluation against a specific `$ai_generation` event | | `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations (no save) | | `posthog:llma-evaluation-summary-create` | AI-powered summary of pass/fail/N/A patterns across runs | | `posthog:execute-sql` | Ad-hoc HogQL over `$ai_evaluation` events | | `posthog:query-llm-trace` | Drill into the underlying generation that an evaluation scored |

All `llma-evaluation-*` tools are defined in `products/ai_observability/mcp/tools.yaml`.

Event schema

Every run of an evaluation emits an `$ai_evaluation` event. Key properties:

| Property | Meaning | | ---------------------------- | --------------------------------------------------------------- | | `$ai_evaluation_id` | UUID of the evaluation config | | `$ai_evaluation_name` | Human-readable name | | `$ai_target_event_id` | UUID of the `$ai_generation` event being scored | | `$ai_trace_id` | Parent trace ID (for jumping to the trace UI) | | `$ai_evaluation_result_type` | Result kind: `boolean` or `sentiment` | | `$ai_evaluation_result` | For boolean evaluations: `true` = pass, `false` = fail | | `$ai_evaluation_reasoning` | Free-text explanation (set by the LLM judge or Hog code) | | `$ai_evaluation_applicable` | `false` when the evaluator decided the generation is N/A | | `$ai_sentiment_label` | For sentiment evaluations: `positive`, `neutral`, or `negative` | | `$ai_sentiment_score` | Confidence score for the winning sentiment label |

When `$ai_evaluation_applicable = false`, the run counts as N/A regardless of `$ai_evaluation_result`. For evaluations that don't support N/A, this property may be `null` — treat null as "applicable".

Workflow: investigate why an evaluation is failing

Works the same way for boolean `llm_judge` and `hog` evaluations — the differences only matter when you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source). Sentiment evaluations should be inspected by sentiment label and score rather than pass/fail filters.

Step 1 — Find the evaluation

posthog:llma-evaluation-list
{ "search": "hallucination", "enabled": true }

Look at the returned `id`, `name`, `evaluation_type`, and either:

• `evaluation_config.prompt` for an `llm_judge`
• `evaluation_config.source` for a `hog` evaluator

The Hog source is the ground truth for why a hog evaluator passes or fails — read it before assuming the failure is in the generation.

Step 2 — Get the AI-generated summary

posthog:llma-evaluation-summary-create
{
  "evaluation_id": "<uuid>",
  "filter": "fail"
}

Returns:

• `overall_assessment` — natural-language summary
• `fail_patterns` — grouped patterns with `title`, `desc