/creating-online-evaluations

name: creating-online-evaluations
description: >
  Author continuously-running online evaluations in PostHog AI observability, grounded in real failure
  modes you've identified. Use when the user wants evaluations that automatically score new generations or
  whole traces going forward — "create an eval to catch X", "continuously check that responses do Y",
  "turn these failures into evals", or after reading traces finds a recurring failure. Covers letting the explored data decide how many evals to create,
  proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge /
  sentiment), configuring an llm_judge's provider and model, scoping which generations trigger it via conditions, creating disabled, verifying scope,
  enabling, and backfilling past data. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the
  failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage
  evaluations that already exist, use exploring-llm-evaluations.

Creating online evaluations

An **online evaluation** automatically scores either each matching `$ai_generation` or the whole trace containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic, not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once those failure modes are identified and turns them into scoped, continuously-running evals.

**One eval per failure mode, and as many evals as the data justifies.** How many to create is a judgment call you make from what the traces actually showed — sometimes one, often three or four. Never assume the answer is one, and never bundle unrelated modes into one evaluator. The exception is a set of modes that are outcomes of the same check, such as the reasons a reply missed the user's request: one categorical eval with a category per mode fits that (2.1).

**Propose before you create.** Bring the user a short list of candidate evals and let them pick which ones they want (Phase 1.1). Creating evals they didn't ask for costs them money and noise.

**First, know what you're evaluating.** Finding and ranking the failure modes worth catching is a separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague about it and don't refer to a specific failure mode, run `exploring-ai-failures` to scope a use case, find failing traces, and produce a ranked list of failure modes.

For the mechanics of _writing and iterating_ an evaluator (Hog source vs LLM-judge prompt, dry-running, debugging a live eval), defer to `exploring-llm-evaluations`.

Tools

| Tool | Purpose | | ------------------------------------------- | ------------------------------------------------------------- | | `posthog:llma-evaluation-config-get` | Check the active provider key used by unpinned judges | | `posthog:llma-provider-key-list` | Find a usable (`ok` state) provider key to pin | | `posthog:llma-evaluation-judge-models` | List valid provider+model combos | | `posthog:llma-evaluation-directory-list` | List directories available for organizing the evaluation | | `posthog:llma-evaluation-directory-create` | Create a directory when the user asks for a new one | | `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations before creating | | `posthog:llma-evaluation-create` | Create the evaluation (always `enabled: false` first) | | `posthog:llma-evaluation-run` | Spot-run a draft eval against one generation | | `posthog:llma-evaluation-update` | Iterate config, then flip `enabled: true` | | `posthog:llma-evaluation-backfill-estimate` | Count what a backfill over past data would evaluate | | `posthog:llma-evaluation-backfill-create` | Evaluate past data, only after the user confirms the estimate | | `posthog:llma-evaluation-backfill-get` | Track a backfill's progress and coverage | | `posthog:llma-evaluation-backfill-list` | Find an evaluation's backfills, including one already running | | `posthog:llma-evaluation-backfill-cancel` | Stop a running backfill when the user asks | | `posthog:execute-sql` | Verify a condition matches the events and volume you expect | | `posthog:generate-app-url` | Build a region- and project-qualified deep link to the eval |

The full create payload (every field, the config schemas, the exact `conditions` shape) is in [references/evaluation-payload.md](references/evaluation-payload.md).

Phase 1 — Decide what to propose, then let the user choose

Start from real, observed failures, not metrics you picked in advance. If you don't already have them, run `exploring-ai-failures` to scope a use case, find failing traces, and produce a ranked list of failure modes — then come back.

1.1 — Turn the failure modes into a candidate set

**Let the data decide how many.** One failure mode is one eval, so a ranked list of four distinct modes is a candidate set of four evals. Don't collapse them into one evaluator that tries to catch everything, and don't stop at the top mode when the traces clearly showed more worth watching. Keep a candidate when:

• **It hurts.** Frequent or painful. A handful of modes usually covers the majority of failures.
• **It's checkable.** It reduces to one crisp criterion — "the reply must stay on the user's topic", "the

tool call must include an `order_id`". If you can't state it in a line, it isn't ready to propose.

• **It's distinct.** Two candidates that would fail on the same generations are one eval.

Rank by how much they hurt and propose roughly