/eval

name: eval
user-invocable: true
tags: [eval, measurement, quality, meta, standard]
model: sonnet
model-preference: sonnet
model-preference-codex: gpt-5.4-mini
model-preference-cursor: claude-sonnet-4-6
args-schema:
  - flag: --session
    description: "session_id to evaluate (default: last completed session via the resolution cascade)"
  - flag: --no-write
    description: "Evaluate + report without appending to the eval journal (.orchestrator/metrics/eval.jsonl)"
  - flag: --verify
    description: "Re-evaluate a stored run-id and diff per-dimension for scoring drift (exit 1 on drift)"
description: >
  Use this skill to run an honest session-process evaluation (Standard v1, aiat-llm-eval/1.0) — score the last completed orchestrator session against the pre-registered rubric-v1 dimensions, run /eval, evaluate this session, produce an eval report, or re-verify a stored eval run for reproducibility. Deterministic-first with an optional advisory LLM judge; never produces a global score.

> **Platform Note:** State files use the platform's native directory: `.claude/` (Claude Code), `.codex/` (Codex CLI), or `.cursor/` (Cursor IDE). Shared metrics + the eval journal live in `.orchestrator/metrics/`. See `skills/_shared/platform-tools.md`.

Eval Skill — Session-Process Evaluation (aiat-llm-eval/1.0)

On-demand, honest measurement of ONE completed orchestrator session against the pre-registered **rubric-v1** check set. The deterministic engine (`scripts/eval-session.mjs` → `scripts/lib/eval/engine.mjs`) reads only local metrics files (`sessions.jsonl` + `events.jsonl`), scores the five deterministic dimensions, appends a `session-eval` record to the journal, and optionally renders an HTML report. An opt-in LLM judge overlays two advisory dimensions.

The standard this skill implements is [`docs/eval/aiat-llm-eval-v1.md`](../../docs/eval/aiat-llm-eval-v1.md); the frozen, content-hashed check set is [`skills/eval/rubric-v1.md`](./rubric-v1.md).

Posture Contract (load-bearing — read before executing)

• **No global score, by construction.** The record has no overall/total/mean

field, and this skill never derives one. Report per-dimension verdicts only.

• **Never guess.** Missing source data yields `cannot-determine` (a first-class,

non-error verdict) with an honest reason — never a fabricated `pass`/`fail`. Do NOT "fill in" a missing KPI or infer a gate result the events do not show.

• **Deterministic before judge.** The five deterministic dimensions are complete

on their own. The judge (Phase 3) is opt-in, ADVISORY, and `uncalibrated` in v1 — never blend a judge verdict into the deterministic tally.

• **Journal is SSOT; the report is a derived view.** The append-only

`.orchestrator/metrics/eval.jsonl` is authoritative. The HTML report is rebuildable from any stored record and is never authoritative over the journal.

• **`--verify` is the reproducibility proof.** Re-scoring stored source data

reproduces the stored dimensions byte-for-byte (exit 0) or reports drift (exit 1). This proves the SCORING replays — NOT that the model is deterministic.

• **Self-evaluation is labelled as such.** The orchestrator scoring its own

session is a self-evaluation, not an independent audit.

---

Phase 0: Bootstrap Gate

Read `skills/_shared/bootstrap-gate.md` and execute the gate check. If the gate is CLOSED, invoke `skills/bootstrap/SKILL.md` and wait for completion before proceeding. If the gate is OPEN, continue to Phase 1.

<HARD-GATE> Do NOT proceed past Phase 0 if GATE_CLOSED. There is no bypass. Refer to `skills/_shared/bootstrap-gate.md` for the full HARD-GATE constraints. </HARD-GATE>

---

Phase 1: Config & Argument Loading

1.1 Read Session Config

Read and parse Session Config per `skills/_shared/config-reading.md`. Extract the `eval` block (`scripts/lib/config.mjs` returns it as `config.eval`, parsed by `scripts/lib/config/eval.mjs`):

enabled:  boolean  (default false)
mode:     'warn' | 'off'            (default 'warn')
judge:    'off' | 'haiku' | 'sonnet' (default 'off')
report:   'html' | 'none'           (default 'html')
handle:   string | null             (default null)

**On-demand `/eval` runs regardless of `eval.enabled`.** The `enabled` flag gates the AUTOMATIC session-end eval phase only — it does NOT gate this command (same posture as `/reconcile` vs `reconcile.enabled`). `mode: off` is honoured as a kill-switch only for the automatic phase; on-demand invocation still runs. If `eval.judge` is `off`, skip Phase 3 entirely.

> **Parser gotcha:** the `eval:` key-line itself MUST NOT carry an inline comment > (strict `/^eval:\s*$/`); a trailing `# comment` on that exact line makes the > parser skip the whole block and silently apply ALL defaults. Sub-key lines > tolerate inline comments.

1.2 Parse Arguments

Inspect `$ARGUMENTS`:

• `--session <id>` → pass through to `--session`.
• `--no-write` → evaluate without appending to the journal (dry-run).
• `--verify <run-id>` → **verification mode**: skip Phases 2–4, run the CLI

`--verify` path (see Phase 6), report MATCH/DRIFT, done.

1.3 Capture the Model Id (honest provenance)

The record's `model.source` records HOW the model id was captured, precisely because self-report is unreliable:

• If `$ANTHROPIC_MODEL` is set in the environment, the engine reads it

automatically with `source: env` — **env wins over the flag** (precedence `env > flag`). Do not pass `--model-id` in that case; let the engine resolve it.

• Otherwise the coordinator passes its own self-reported model id:

`--model-id <self-reported-model-id> --model-source self-report`.

---

Phase 2: Deterministic Run

Run the deterministic engine via its CLI. Default target is the last completed session (resolution cascade); `--session` overrides.

node scripts/eval-session.mjs [--session <id>] --json \
  [--model-id <self-reported-id> --model-source self-report] \
  [--no-write]

• **Do NOT pass `--metrics-