/Assay

name: Assay
metadata:
  short-description: "Evaluates prompts, models and agents with code asserts, calibrated judges, pass@k and A/B tests; designs judgment questions and code-extract-decide-write pipelines."
version: 2.9.3
evals:
  suites: [core-dispositions]
  deterministic:
    - command: "bun LIFEOS/ASSAY/QuestionSystem.ts validate LIFEOS/ASSAY/fixtures/question-system.example.json"
      falsifier: "the clean example question system exits nonzero"
description: "Assertion-first eval framework: code asserts, calibrated judges, pass@k, A/B comparison, agentic evals, judgment-layer question systems and code-extract-decide-write pipelines. USE WHEN eval, evaluate, benchmark, judge, judge prompt, judge panel, regression test, assert, llm-rubric, calibrate judge, judged residue, pass@k, pass^k, grade output, compare prompts or models, A/B test, which model did we like, model retro, ablation, with/without, test agent, agentic eval, noul/choice/score question, glance row, judged miss, proof this, check the output, did it come out as asked, design judgment questions, question system, design a pipeline, extract values, Clairic, LLM-only baseline, judgment layer status/report/tune, is this caller ready to enforce. NOT FOR scientific-method framing (Science), property/mutation testing (Hardening), live UI verification (Interceptor), non-eval prompts (Prompting), TypeSafe apps outside LifeOS (typesafe-ai)."

Assay — Assertion-First AI Evaluation

**Contents:** Ideal State · Intent Routing · The taxonomy · Choosing a check: tier first, then shape · Selection, acceptance, and unavailable evidence · The canonical path · Doctrine (from Anthropic; encode, don't restate) · The image lane (visual evals) · Harness integration · Judgment-layer question systems · Data & Config · Gotchas · Examples

An eval gives an AI an input, then applies **assertions** to its output to measure success (Anthropic's definition). A case is `{id, prompt, assert:[...]}`. Each assertion is either deterministic (code, free) or model-graded (a calibrated judge). Cases run multiple trials and report **pass^k** (every trial passes; the reliability metric and the regression gate) and **pass@k** (any trial passes). Proof applies the same tiers to one run's output (the film, image, page, email or document it made) against what was asked, before the principal sees it: `Intents/Proof.md`, with the whole system in `LIFEOS/DOCUMENTATION/Assay/AssaySystem.md`. CreateQuestions designs the fast tier itself: it turns a fuzzy decision into a measured judgment-layer question system (`LIFEOS/DOCUMENTATION/Assay/AssaySystem.md` § Judgment layer).

Sources: [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (2026-01), [A Statistical Approach to Model Evaluations](https://www.anthropic.com/research/statistical-approach-to-model-evals) (2024-11), [develop tests](https://platform.claude.com/docs/en/docs/build-with-claude/develop-tests), the eval-awareness contamination post (2026-03), and the live `claude plugin eval` surface.

**Freshness contract:** "aligned to Anthropic's doctrine" is a live claim. When designing a new suite class or touching `## Doctrine`, re-fetch the demystifying-evals post and flag where it has moved. Report divergence, never auto-adopt; an unreachable URL never blocks a run.

Ideal State

• Every check sits at the lowest tier that can honestly answer it (code, glance, judge, human), and only code runs on a blocking path.
• Every score ships with pass^k, SEM and a 95% CI from the trials that completed, with the requested and completed trial counts beside it; fewer completed trials than requested never claims pass^k.
• A judged check is calibrated against the principal's labels or reads PROVISIONAL wherever it shows, and a judged suite keeps an adversarial fixture its judge must fail.
• A produced artifact is checked by Proof against what was asked before the principal sees it, and the receipt is bound to its bytes.
• Transport failures, malformed verdicts and missing observations are reported as unavailable evidence, never as a pass, a quality failure or an adversarial catch.
• A fuzzy decision handed to CreateQuestions leaves as a validated question system whose accuracy was measured on held-out cases the designer never saw, with its margin over one holistic baseline question, or as a stated finding that the judgment layer does not fit and what does; a design ends at a saved shadow registry row and never moves a caller toward enforce.
• When Assay builds the eval system for a project, the project's ISA ends current: every criterion it checks names its suite or glance question in Test Strategy, and the hand-rolled eval code, rows and stale criteria it replaced are removed in the same change.

Intent Routing

| Intent | Trigger | File | |----------|---------|------| | **Proof** | "proof this", "check the output", "did it come out as asked", before showing any produced artifact | `Intents/Proof.md` | | **RunEval** | "run the eval", "run suite", "evaluate this" | `Intents/RunEval.md` | | **CreateSuite** | "new eval", "create a suite", "eval for X" | `Intents/CreateSuite.md` | | **CreateJudge** | "write a judge", "llm-rubric", "grading criteria", "noul, choice or score", "glance question" | `Intents/CreateJudge.md` | | **Calibrate** | "calibrate the judge", "judge agreement", "judged residue", "shrink the judges", "judged miss", "bunker miss" | `Intents/Calibrate.md` | | **Compare** | "compare prompts", "compare models", "A/B this", "with/without", "ablation" | `Intents/Compare.md` | | **ModelRetro** | "which model did we like", "model retro", "compare models looking back", "how did model X do historically" | `Intents/ModelRetro.md` | | **RunAgentic** | "multi-turn eval", "agentic eval", "eval with tools", "plugin eval" | `Intents/RunAgentic.md` | | **ViewResults** | "eval results", "how did it score", "saturation" | `Intents/ViewResults.md` | | **CreateQue