/exploring-llm-evaluations

name: exploring-llm-evaluations
description: >
  Investigate AI observability evaluations — `hog` (deterministic code-based),
  `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment).
  Find existing evaluations, inspect their configuration, run them against
  specific generations, query individual results, and set up scheduled reports
  on an evaluation.
  Use when the user asks to debug why an evaluation is failing, surface common
  failure modes, compare results across filters, dry-run a Hog evaluator,
  prototype a new LLM-judge prompt, inspect sentiment classifications, or manage
  the evaluation lifecycle.

Exploring AI observability evaluations

PostHog evaluations score `$ai_generation` events. Each evaluation is one of three types:

• **`hog`** — deterministic Hog code that returns `true`/`false` (and optionally N/A).

Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code.

• **`llm_judge`** — an LLM scores generations against a prompt you write. Best for

subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic drift, instruction-following. Costs an LLM call per run and requires AI data processing approval at the org level.

• **`sentiment`** — classifies sentiment from user messages on each matching

generation. Returns a sentiment label and score, not a pass/fail verdict.

Results from all types land in ClickHouse as `$ai_evaluation` events. Boolean evaluations (`llm_judge` and `hog`) set `$ai_evaluation_result`; sentiment evaluations set `$ai_sentiment_*` properties instead.

This skill covers the full lifecycle: list/inspect/manage evaluation configs, run them on specific generations, query individual results, and configure evaluation reports that summarize recent runs on a schedule.

Tools

| Tool | Purpose | | ----------------------------------------- | -------------------------------------------------------------- | | `posthog:llma-evaluation-list` | List/search evaluation configs (filter by name, enabled flag) | | `posthog:llma-evaluation-get` | Get a single evaluation config by UUID | | `posthog:llma-evaluation-create` | Create a new `llm_judge`, `hog`, or `sentiment` evaluation | | `posthog:llma-evaluation-update` | Update an existing evaluation (name, prompt, enabled, …) | | `posthog:llma-evaluation-delete` | Soft-delete an evaluation | | `posthog:llma-evaluation-run` | Run an evaluation against a specific `$ai_generation` event | | `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations (no save) | | `posthog:llma-evaluation-report-list` | List the report configs attached to an evaluation | | `posthog:llma-evaluation-report-create` | Schedule an AI report on an evaluation (email or Slack) | | `posthog:llma-evaluation-report-run-list` | Past report runs, including the report content that was sent | | `posthog:execute-sql` | Ad-hoc HogQL over `$ai_evaluation` events | | `posthog:query-llm-trace` | Drill into the underlying generation that an evaluation scored |

All `llma-evaluation-*` tools are defined in `products/ai_observability/mcp/tools.yaml`.

Event schema

Every run of an evaluation emits an `$ai_evaluation` event. Key properties:

| Property | Meaning | | ---------------------------- | -------------------------------------------------------------------------------- | | `$ai_evaluation_id` | UUID of the evaluation config | | `$ai_evaluation_name` | Human-readable name | | `$ai_target_event_id` | UUID of the `$ai_generation` event being scored | | `$ai_trace_id` | Parent trace ID (for jumping to the trace UI) | | `$ai_evaluation_result_type` | Result kind: `boolean` or `sentiment` | | `$ai_evaluation_result` | Raw boolean result. Use the evaluation's output config to map it to pass or fail | | `$ai_evaluation_reasoning` | Free-text explanation (set by the LLM judge or Hog code) | | `$ai_evaluation_applicable` | `false` when the evaluator decided the generation is N/A | | `$ai_sentiment_label` | For sentiment evaluations: `positive`, `neutral`, or `negative` | | `$ai_sentiment_score` | Confidence score for the winning sentiment label |

When `$ai_evaluation_applicable = false`, the run counts as N/A regardless of `$ai_evaluation_result`. For evaluations that don't support N/A, this property may be `null` — treat null as "applicable". For boolean evaluations, `output_config.true_is_failure: false` maps `true` to pass and `false` to fail. Set it to `true` for detector-style evaluations where `true` means the evaluator found a problem.

Workflow: investigate why an evaluation is failing

Works the same way for boolean `llm_judge` and `hog` evaluations — the differences only matter when you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source). Sentiment evaluations should be inspected by sentiment label and score rather than pass/fail filters.

Step 1 — Find the evaluation

posthog:llma-evaluation-list