/creating-online-evaluations

name: creating-online-evaluations
description: >
  Author continuously-running online evaluations in PostHog AI observability, grounded in real failure
  modes you've identified. Use when the user wants evaluations that automatically score new generations or
  whole traces going forward — "create an eval to catch X", "continuously check that responses do Y",
  "turn these failures into evals". Covers letting the explored data decide how many evals to create,
  proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge /
  sentiment), configuring a provider and model for an llm_judge eval (a provider key gates enabling, not
  creation), scoping which generations trigger it via conditions, creating disabled, verifying scope, and
  enabling. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the
  failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage
  evaluations that already exist, use exploring-llm-evaluations.

Creating online evaluations

An **online evaluation** automatically scores either each matching `$ai_generation` or the whole trace containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic, not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once those failure modes are identified and turns them into scoped, continuously-running evals.

**One eval per failure mode, and as many evals as the data justifies.** How many to create is a judgment call you make from what the traces actually showed — sometimes one, often three or four. Never assume the answer is one, and never bundle several modes into one evaluator.

**Propose before you create.** Bring the user a short list of candidate evals and let them pick which ones they want (Phase 1.1). Creating evals they didn't ask for costs them money and noise.

**First, know what you're evaluating.** Finding and ranking the failure modes worth catching is a separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague about it and don't refer to a specific failure mode, run `exploring-ai-failures` to scope a use case, find failing traces, and produce a ranked list of failure modes.

For the mechanics of _writing and iterating_ an evaluator (Hog source vs LLM-judge prompt, dry-running, debugging a live eval), defer to `exploring-llm-evaluations`.

Tools

| Tool | Purpose | | ------------------------------------------ | ------------------------------------------------------------- | | `posthog:llma-evaluation-config-get` | Check the active provider key used by unpinned judges | | `posthog:llma-provider-key-list` | Find a usable (`ok` state) provider key to pin | | `posthog:llma-evaluation-judge-models` | List valid provider+model combos | | `posthog:llma-evaluation-directory-list` | List directories available for organizing the evaluation | | `posthog:llma-evaluation-directory-create` | Create a directory when the user asks for a new one | | `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations before creating | | `posthog:llma-evaluation-create` | Create the evaluation (always `enabled: false` first) | | `posthog:llma-evaluation-run` | Spot-run a draft eval against one generation | | `posthog:llma-evaluation-update` | Iterate config, then flip `enabled: true` | | `posthog:execute-sql` | Verify a condition matches the events and volume you expect | | `posthog:generate-app-url` | Build a region- and project-qualified deep link to the eval |

The full create payload (every field, the config schemas, the exact `conditions` shape) is in [references/evaluation-payload.md](references/evaluation-payload.md).

Phase 1 — Decide what to propose, then let the user choose

Start from real, observed failures, not metrics you picked in advance. If you don't already have them, run `exploring-ai-failures` to scope a use case, find failing traces, and produce a ranked list of failure modes — then come back.

1.1 — Turn the failure modes into a candidate set

**Let the data decide how many.** One failure mode is one eval, so a ranked list of four distinct modes is a candidate set of four evals. Don't collapse them into one evaluator that tries to catch everything, and don't stop at the top mode when the traces clearly showed more worth watching. Keep a candidate when:

• **It hurts.** Frequent or painful. A handful of modes usually covers the majority of failures.
• **It's checkable.** It reduces to one crisp criterion — "the reply must stay on the user's topic", "the

tool call must include an `order_id`". If you can't state it in a line, it isn't ready to propose.

• **It's distinct.** Two candidates that would fail on the same generations are one eval.

Rank by how much they hurt and propose roughly the top five; mention in a line that you set weaker ones aside rather than silently dropping them.

1.2 — Propose the set in plain language

**Never create evals the user hasn't picked.** Lay out the candidates and ask which ones they want.

Assume they haven't read the traces with you and don't know the eval vocabulary. Keep each one to a line or two — a wall of text per eval means they can't compare them — but make it obvious what it watches and what they'll see when it fails. No Hog snippets, property filters, or `hog`/`llm_judge` internals here. Number them so they can reply "1 and 3":

> Found 3 failure patterns worth watching. Which should I set up? > > **1. Replies drift off topic** — checks the answer addresses what the user actually asked. > Catches suppo