/surrogate-verifier

name: surrogate-verifier
description: 'Generates structured test assertions and failure diagnostics for skill packages from a definition and task prompt. Triggers on: "verify this skill", "generate assertions", "surrogate verification", "diagnose skill failure". NOT for code review, use pr-review.'
metadata:
  version: 1.0.1
  category: review
  tags: [verification, assertions, testing, co-evolution, diagnostics, eval]
  difficulty: advanced
  phase: verify

Surrogate Verifier

Generate structured test assertions and failure diagnostics for skill packages through information-isolated verification. The verifier operates without access to the skill generator's reasoning — it sees only the skill definition, a task prompt, and the output artifacts. This isolation prevents confirmation bias and is the single largest contributor to skill quality in co-evolutionary generation (+30pp per EvoSkills).

Reference Files

| File | Contents | Load When | | ----------------------------------- | ---------------------------------------------------------------- | ------------------------------ | | `references/assertion-patterns.md` | Assertion catalog by skill category with weight guidance | Always | | `references/diagnostic-templates.md`| Failure diagnostic templates with root-cause categories | When producing failure reports |

Information Isolation Protocol

This is the most critical constraint. Violating isolation degrades verification quality.

**The verifier MUST NOT access:**

• The generator's conversation history or reasoning chain
• Prior evolution iterations or refinement context
• The generator's internal notes or decision rationale
• Any context beyond what is explicitly listed below

**The verifier receives ONLY:** 1. The skill's `SKILL.md` content (the definition file) 2. One or more task prompts representing intended use 3. The skill's output (when diagnosing failures) 4. The assertion results from `scripts/eval_assertions.py` (when diagnosing)

**Implementation:** When invoked by the `test-engineer` agent, this skill MUST be loaded into a **separate Agent spawn** using `isolation: "worktree"` or at minimum a fresh session with no shared context. The invoking agent passes artifacts as explicit text, not as conversation references.

Workflow

Mode 1: Assertion Generation

Generate assertions for a skill given its definition and task prompts.

Phase 1: Skill Analysis

Read the `SKILL.md` definition and extract:

1. **Stated capabilities** — what the skill claims to do (from description + workflow sections) 2. **Output format** — expected structure of the skill's output (markdown, JSON, tables, etc.) 3. **Error handling** — documented failure modes and recovery paths 4. **Prerequisites** — required tools, dependencies, or context 5. **Trigger boundaries** — what the skill does NOT handle (negative scope)

Phase 2: Assertion Design

For each task prompt, generate 5-10 assertions covering these dimensions:

| Dimension | Assertion Types to Use | Purpose | | --------------------- | ------------------------------- | ------------------------------------------ | | Output completeness | `contains`, `matches_regex` | All claimed sections/components present | | Format compliance | `output_format`, `contains` | Output matches declared structure | | Factual signals | `contains`, `not_contains` | Key domain terms present, hallmarks absent | | Tool usage | `calls_tool` | Expected tools were invoked | | Negative constraints | `not_contains` | Forbidden patterns absent |

**Weight assignment:**

• Output completeness assertions: weight 1.0 (must have)
• Format compliance: weight 0.8 (structural correctness)
• Factual signals: weight 0.6 (content quality)
• Tool usage: weight 0.5 (method verification)
• Negative constraints: weight 0.3 (absence checks are weaker signals)

See `references/assertion-patterns.md` for category-specific assertion catalogs.

Phase 3: Output

Produce assertions in the `evals/cases.yaml` schema format:

assertions:
  - type: contains
    target: "## Scalability"
    weight: 1.0
  - type: output_format
    target: markdown_table
    weight: 0.8
  - type: not_contains
    target: "TODO"
    weight: 0.3
  - type: calls_tool
    target: Read
    weight: 0.5

**Context cap:** Do not consume more than 70% of the available context window. If the skill definition is very long, focus assertion generation on the workflow phases and output format sections. Summarize rather than quote verbatim.

Mode 2: Failure Diagnostics

When an oracle returns `fail`, produce a structured diagnostic explaining why.

Input

• The skill's `SKILL.md` (same as Mode 1)
• The task prompt that was executed
• The output that failed
• The assertion results: which passed, which failed, with details

Phase 1: Failure Classification

Categorize each failed assertion into a root-cause category:

| Category | Signal | Severity | | --------------------- | -------------------------------------------------------------- | ---------- | | Missing capability | `contains` assertion failed for a claimed feature | HIGH | | Format mismatch | `output_format` assertion failed | HIGH | | Incomplete output | Multiple `contains` assertions failed in the same section | MEDIUM | | Hallucinated content | `not_contains` assertion failed (forbidden pattern present) | HIGH | | Wrong tool usage | `calls_tool` assertion failed | MEDIUM | | Partial success | Some