/nw-agent-evals

name: nw-agent-evals
description: Lightweight eval method for testing nWave AGENTS and SKILLS (LLM behavior) as a lean alternative to heavy BDD/ATD. An eval = one prompt -> one captured run (trace + artifacts) -> a small set of checks -> a comparable score over time. Load when validating agent behavior, building a regression net for an agent/skill, or reducing agent-test bloat.
user-invocable: false
disable-model-invocation: true

Agent Evals

Why this exists

Agents are LLMs — non-deterministic. Traditional BDD/unit testing tests them *badly* and breeds bloat (measured: 294K test LOC, 5.5:1 test:src, step reuse 1.10x vs >=4x target). Evals are the lean way to verify agent behavior: a few targeted signals instead of a monolithic spec.

This skill is the **antithesis of ATD over-specification**. Keep it small. The anti-pattern is up-front exhaustive specification — that IS the bloat.

| | `nw-agent-testing` (sibling skill) | `nw-agent-evals` (this skill) | |---|---|---| | Form | static 5-layer manual checklist | executable dataset + grader + score over time | | Use | one-shot design review of a spec | repeatable regression net for behavior | | Output | pass/fail judgement | comparable score, trend across runs |

Use both: `nw-agent-testing` to vet the spec, `nw-agent-evals` to watch behavior over time.

What an eval is

One eval = **prompt -> run -> checks -> score**.

• **prompt** — a single input that should (or should NOT) trigger the agent/skill.
• **run** — one provider turn through the selected nWave adapter, with its

available native evidence and artifacts captured.

• **checks** — a small set of targeted assertions (not one monolithic check).
• **score** — a comparable number you can track across runs to catch regressions.

Replaces "vibes" with measurable signals: *did it invoke the right skill, run the expected tools, respect the conventions and produce the required observable effect?* Never grade a terminal-text grammar as a behavioral outcome.

Definition of Done — before you write the eval

Write the success criteria FIRST, before implementing the agent/skill or its eval. Four check categories:

| Category | Question | Graded by | |---|---|---| | OUTCOME | Did the task complete through its required public effect and evidenced semantic result? | deterministic | | PROCESS | Was required skill delivery or use evidenced, with only behavior-relevant tool observations? | deterministic | | STYLE | Does the output respect nWave conventions (sections, format)? | model-graded | | EFFICIENCY | No useless commands / no token blowup? | deterministic |

If you cannot state DoD before writing the skill, the skill's job is not yet defined — stop and define it.

Workflow

Run these steps in order:

1. **Define success first** — write a small set of falsifiable checks in the four categories above. 2. **Use one representative probe when needed** — capture the provider-native evidence that the selected adapter actually exposes. 3. **Add only the cases needed** — include a positive case and a relevant negative control; grow the set from observed failures. 4. **Grade deterministic evidence** — check public effects, admitted tool evidence, and structured artifacts. Do not derive a verdict from terminal prose. 5. **Grade semantic quality separately** — use the existing rubric facility; retain narrative feedback as diagnostic evidence, not terminal control input.

Capturing provider evidence

The selected adapter owns the provider projection. Claude roles run through the Claude adapter; Codex roles run through the Codex adapter's `codex exec` projection. Do not substitute one provider's trace shape, tool names, or launcher for the other's.

Distinguish these three facts when evaluating a skill:

| Fact | Evidence | What it establishes | |---|---|---| | **Preloaded knowledge** | The role declares the skill in frontmatter and `load_role_instructions` materializes it in the provider instruction. | The role received the skill before the turn. It creates no native `Read` event. | | **Native read** | A provider-native tool event reads a conditional `SKILL.md` path. Claude's existing transcript tracker records this form. | The turn fetched that file on demand. It does not prove that the knowledge affected the result. | | **Actual use** | A predicted, observable effect or a capability-specific tool/result is present, with no simpler explanation. | The role applied the skill sufficiently for this eval. A catalog entry, preload, or `Read` alone is insufficient. |

Evaluate semantic results and public effects separately from their presentation. Terminal prose, stdout, stderr, recorder entries and native tool traces carry diagnostic/provenance evidence, not interchangeable proof of a durable effect. Use the selected host's native transcript/tool receipts rather than another parser. Inspect only events and recorder evidence that its adapter preserves. If the required observation is absent on the selected provider, report that check as **INDETERMINATE**; do not infer it from another projection.

Check an artifact on disk as well as its reported tool event. Only ask for a tool when the behavior requires it. Tool order and an exact command sequence are not general quality signals.

Deterministic graders (nWave-native signals)

Parse the trace, assert mechanically. nWave-specific, high-value signals:

| Signal | Assertion | Why it matters | |---|---|---| | Declared preload | resolved role instruction contains the declared skill once | verifies delivery of resident knowledge, not its use | | Conditional read | native `Read` event for the resolved skill path, where the provider exposes one | verifies on-demand fetch, not semantic use | | Capability-specific behavior | required public effect or justified tool/result is present | tests actual application of the skill | | Semantic outcome observed | the semantic result and required public effect are evidenced | separates judg