/eval

name: eval
user-invocable: true
argument-hint: "[--session <id>] [--no-write] [--verify <run-id>]"
tags: [eval, measurement, quality, meta, standard]
model: sonnet
model-preference: sonnet
model-preference-codex: gpt-5.4-mini
model-preference-cursor: claude-sonnet-4-6
args-schema:
  - flag: --session
    description: "session_id to evaluate (default: last completed session via the resolution cascade)"
  - flag: --no-write
    description: "Evaluate + report without appending to the eval journal (.orchestrator/metrics/eval.jsonl)"
  - flag: --verify
    description: "Re-evaluate a stored run-id and diff per-dimension for scoring drift (exit 1 on drift)"
description: >
  Use this skill to run an honest session-process evaluation (Standard v1, aiat-llm-eval/1.0) — score the last completed orchestrator session against the pre-registered rubric-v3 dimensions, run /eval, evaluate this session, produce an eval report, or re-verify a stored eval run for reproducibility. Deterministic-first with an optional advisory LLM judge; never produces a global score.

> **Platform Note:** State files use the platform's native directory: `.claude/` (Claude Code), `.codex/` (Codex CLI), or `.cursor/` (Cursor IDE). Shared metrics + the eval journal live in `.orchestrator/metrics/`. See `skills/_shared/platform-tools.md`.

Eval Skill — Session-Process Evaluation (aiat-llm-eval/1.0)

On-demand, honest measurement of ONE completed orchestrator session against the pre-registered **rubric-v3** check set. The deterministic engine (`scripts/eval-session.mjs` → `scripts/lib/eval/engine.mjs`) reads only local metrics files (`sessions.jsonl` + `events.jsonl`), scores the six deterministic dimensions, appends a `session-eval` record to the journal, and optionally renders an HTML report. An opt-in LLM judge overlays ONE advisory dimension (`instruction-adherence`; `report-quality` was retired in rubric-v2, #1381).

The standard this skill implements is [`docs/eval/aiat-llm-eval-v1.md`](../../docs/eval/aiat-llm-eval-v1.md); the frozen, content-hashed check set is [`skills/eval/rubric-v3.md`](./rubric-v3.md) (stored records carry the version they were scored under: before 2026-09-19 `rubric-v1` → [`rubric-v1.md`](./rubric-v1.md), 2026-09-19 to 2026-10-05 `rubric-v2` → [`rubric-v2.md`](./rubric-v2.md); neither is edited again).

Invocation

Invoked as `/eval [--session <id>] [--no-write] [--verify <run-id>]` with arguments: **$ARGUMENTS** (parsed in Phase 1.2).

• `/eval` — evaluate the last completed session (resolution cascade), append the record, render the report.
• `/eval --session <id>` — evaluate a specific `session_id`.
• `/eval --no-write` — evaluate + report without appending to the journal (dry-run).
• `/eval --verify <run-id>` — re-score a stored run and diff for drift (the reproducibility proof; exit 1 on drift).

**On-demand `/eval` runs regardless of `eval.enabled`** — that flag gates only the automatic session-end eval phase (see Phase 1.1).

**Seams used:** `scripts/eval-session.mjs` (deterministic CLI) · `runEvalJudge` / `mergeJudgeDimensions` (`scripts/lib/eval/judge.mjs`, opt-in) · `writeEvalReport` (`scripts/lib/eval/report.mjs`) · `appendEvalRecord` (`scripts/lib/eval/sink.mjs`) · the `eval` config block · [`skills/eval/rubric-v3.md`](./rubric-v3.md) (frozen check set).

Posture Contract (load-bearing — read before executing)

• **No global score, by construction.** The record has no overall/total/mean

field, and this skill never derives one. Report per-dimension verdicts only.

• **Never guess.** Missing source data yields `cannot-determine` (a first-class,

non-error verdict) with an honest reason — never a fabricated `pass`/`fail`. Do NOT "fill in" a missing KPI or infer a gate result the events do not show.

• **Deterministic before judge.** The six deterministic dimensions are complete

on their own. The judge (Phase 3) is opt-in, ADVISORY, and `uncalibrated` — never blend a judge verdict into the deterministic tally.

• **Journal is SSOT; the report is a derived view.** The append-only

`.orchestrator/metrics/eval.jsonl` is authoritative. The HTML report is rebuildable from any stored record and is never authoritative over the journal.

• **`--verify` is the reproducibility proof.** Re-scoring stored source data

reproduces the stored dimensions byte-for-byte (exit 0) or reports drift (exit 1). This proves the SCORING replays — NOT that the model is deterministic.

• **Self-evaluation is labelled as such.** The orchestrator scoring its own

session is a self-evaluation, not an independent audit.

---

Phase 0: Bootstrap Gate

Read `skills/_shared/bootstrap-gate.md` and execute the gate check. If the gate is CLOSED, invoke `skills/bootstrap/SKILL.md` and wait for completion before proceeding. If the gate is OPEN, continue to Phase 1.

<HARD-GATE> Do NOT proceed past Phase 0 if GATE_CLOSED. There is no bypass. Refer to `skills/_shared/bootstrap-gate.md` for the full HARD-GATE constraints. </HARD-GATE>

---

Phase 1: Config & Argument Loading

1.1 Read Session Config

Read and parse Session Config per `skills/_shared/config-reading.md`. Extract the `eval` block (`scripts/lib/config.mjs` returns it as `config.eval`, parsed by `scripts/lib/config/eval.mjs`):

enabled:  boolean  (default false)
mode:     'warn' | 'off'            (default 'warn')
judge:    'off' | 'haiku' | 'sonnet' (default 'off')
report:   'html' | 'none'           (default 'html')
handle:   string | null             (default null)

**On-demand `/eval` runs regardless of `eval.enabled`.** The `enabled` flag gates the AUTOMATIC session-end eval phase only — it does NOT gate this command (same posture as `/reconcile` vs `reconcile.enabled`). `mode: off` is honoured as a kill-switch only for the automatic phase; on-demand invocation still runs. If `eval.judge` is `off`, skip Phase 3 entirely.

> **Parser gotcha:** the `eval:` key-line itself MUST NOT carry an inline comment > (strict `/^eval:\