/exploring-llm-evaluations

name: exploring-llm-evaluations
description: >
  Investigate AI observability evaluations — `hog` (deterministic code-based),
  `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment).
  Find existing evaluations, inspect their configuration, run them against
  specific generations, query individual results, and set up scheduled reports
  on an evaluation.
  Use when the user asks to debug why an evaluation is failing, surface common
  failure modes, compare results across filters, dry-run a Hog evaluator,
  prototype a new LLM-judge prompt, inspect sentiment classifications, or manage
  the evaluation lifecycle.

Exploring AI observability evaluations

For stored offline experiment comparisons, scorer history, or externally computed result uploads, use `analyzing-offline-evaluations`. Offline experiments use pinned scorer configurations and separate result APIs from the online evaluations below.

PostHog evaluations score `$ai_generation` events. Each evaluation is one of three types:

• **`hog`** — deterministic Hog code that returns a boolean, numeric score, or categories (and optionally N/A).

Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code.

• **`llm_judge`** — an LLM scores generations against a prompt you write. Best for

subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic drift, instruction-following. Costs an LLM call per run and requires AI data processing approval at the org level.

• **`sentiment`** — classifies sentiment from user messages on each matching

generation. Returns a sentiment label and score, not a pass/fail verdict.

Results from all types land in ClickHouse as `$ai_evaluation` events. Boolean evaluations (`llm_judge` and `hog`) set `$ai_evaluation_result`; sentiment evaluations set `$ai_sentiment_*` properties instead. Both `hog` and `llm_judge` also support `output_type: "numeric"`. Numeric runs store their raw score in `$ai_evaluation_numeric_result`, with optional `$ai_evaluation_numeric_result_min` and `$ai_evaluation_numeric_result_max`. They never set `$ai_evaluation_result`. Use `output_config.passing_rule` to interpret scores: `gte` means at least the threshold and `lte` means at most. Changing the rule reinterprets historical scores. Saved reports retain the rule and metrics used when generated. Without a passing rule, inspect score means and distributions; pass rates and reports are unavailable.

Both runtimes also support `output_type: "categorical"` when categorical evaluations are enabled for the project. Read `output_config.options` for the configured `{key, label}` pairs and `selection_mode` for `single` or `multiple`. Results store keys as a list in `$ai_evaluation_categorical_result`; single selection requires exactly one key, while multiple selection accepts `[]`. Hog can also return a single key string in single mode. Use `output_config.passing_rule.categories` to grade results: a nonempty result passes only when every returned key is a passing category. A nonempty rule fails `[]`; an empty rule passes only `[]` and is allowed only in multiple mode. Without a rule, inspect category frequencies instead of pass rates; new reports are unavailable. Rule edits reclassify historical results, while saved reports retain their rule snapshot.

This skill covers the full lifecycle: list/inspect/manage evaluation configs, run them on specific generations, query individual results, and configure evaluation reports that summarize recent runs on a schedule.

Tools

| Tool | Purpose | | ----------------------------------------- | -------------------------------------------------------------- | | `posthog:llma-evaluation-list` | List/search evaluation configs (filter by name, enabled flag) | | `posthog:llma-evaluation-get` | Get a single evaluation config by UUID | | `posthog:llma-evaluation-create` | Create a new `llm_judge`, `hog`, or `sentiment` evaluation | | `posthog:llma-evaluation-update` | Update an existing evaluation (name, prompt, enabled, …) | | `posthog:llma-evaluation-delete` | Soft-delete an evaluation | | `posthog:llma-evaluation-run` | Run an evaluation against a specific `$ai_generation` event | | `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations (no save) | | `posthog:llma-evaluation-report-list` | List the report configs attached to an evaluation | | `posthog:llma-evaluation-report-create` | Schedule an AI report on an evaluation (email or Slack) | | `posthog:llma-evaluation-report-run-list` | Past report runs, including the report content that was sent | | `posthog:execute-sql` | Ad-hoc HogQL over `$ai_evaluation` events | | `posthog:query-llm-trace` | Drill into the underlying generation that an evaluation scored |

All `llma-evaluation-*` tools are defined in `products/ai_observability/mcp/tools.yaml`.

Event schema

Every run of an evaluation emits an `$ai_evaluation` event. Key properties:

| Property | Meaning | | ----------------------------------- | -------------------------------------------------------------------------------- | | `$ai_evaluation_id` | UUID of the evaluation config | | `$ai_evaluation_name` | Human-readable name