/evals-clarify

name: evals-clarify
description: Use when draft eval criteria need refining, clustering, and acceptance into the published goldset with an isolated holdout split (goldset.md + goldset.json).
disable-model-invocation: true

evals-clarify

What this skill does

Conducts **axial coding** following **EDD Principles III & IX** to cluster related failure patterns, refine evaluation criteria, generate adversarial examples, and accept validated drafts into the published goldset.

**Output**: 1. **Clustered Criteria** - Related patterns grouped into coherent evaluation themes 2. **Adversarial Examples** - Generated attack scenarios and edge cases for robustness 3. **Published Goldset** - Accepted criteria in `evals/{system}/goldset.md` with full documentation 4. **Holdout Dataset** - Reserved test set (20%) for unbiased evaluation validation 5. **JSON Configuration** - Auto-generated `goldset.json` for system consumption 6. **Auto-handoff** to `/evals-implement` for grader generation

**Key EDD Principles Applied**:

• **Principle III**: Error Analysis & Pattern Discovery - Axial coding → theoretical relationships
• **Principle IX**: Test Data as Code - Adversarial generation, holdout splits, version control
• **Principle II**: Binary Pass/Fail - Maintain strict binary evaluation throughout
• **Principle I**: Spec-Driven Contracts - Criteria validate spec compliance

When to use

• **After `/evals-specify`**: Refine and accept draft criteria into goldset
• **Dataset maintenance**: Balance pass/fail examples or add adversarial cases
• **Adding holdout split**: Isolate validation data from training data

When NOT to use

• **No draft criteria exist**: Run `/evals-specify` to discover patterns first
• **Grader generation**: Use `/evals-implement` to convert accepted goldset into code

Process

User Input

$ARGUMENTS

• `--accept IDS` — Accept specific draft IDs (e.g., "EVAL-001,EVAL-003")
• `--merge IDS` — Merge related criteria (e.g., "EVAL-001+EVAL-002")
• `--split ID` — Split complex criterion into multiple focused criteria
• `--holdout-ratio RATIO` — Holdout percentage (default: 0.2, range: 0.1-0.3)

Execution Steps

Step 0: Detect Lightweight Draft Format

Check if the draft being reviewed uses the lightweight draft template (indicated by presence of `type`, `evidence`, `source`, `revisit-when` fields in frontmatter and `## Rejected Alternatives` / `## Reason` body sections without the full formal template sections).

If lightweight: 1. Read the draft's captured fields (Context, Decision, Rejected Alternatives, Reason) 2. Transform to full formal template:

• ADR: full MADR format with Decision Drivers, Considered Options, Pros/Cons, Constitution Alignment, Related ADRs
• PDR: full PDR format with Market Forces, Consequences, Alternatives Considered, Links
• ChDR: full ChDR format with Issue Links, Commits, Consequences, Evidence
• CDR: full CDR format with Context Type, Target Module, Descriptor, Evidence
• EVAL: full eval format with Error Analysis, Pass/Fail Examples, Implementation Notes

3. Enrich from session context (add details the lightweight draft may have omitted) 4. Present the enriched draft for review

If already full format, proceed with normal review.

Phase 1: Axial Coding & Clustering

• Group related draft patterns into coherent themes.
• Resolve any overlaps or duplicate criteria.

Phase 2: Refinement & Adversarial Generation

• Generate 3-5 adversarial (attack) examples per criterion to test robustness.
• Balance pass/fail examples (~50/50 ratio).

Phase 3: Holdout Isolation

• Isolate exactly 20% of examples as a reserved holdout set (saved to `.adlc/memory/evals/holdout.json` — machine data stays under `.adlc/` per the shared layout).
• Ensure holdout set is never used in implementation or training.

Phase 4: Publish Goldset

• Copy accepted drafts to `docs/adlc/memory/evals/` (the shared layout memory root; `holdout.json` alone stays at `.adlc/memory/evals/holdout.json`) and update status to `accepted`.
• Compile published goldset to `evals/{system}/goldset.md` (human-readable) and `evals/{system}/goldset.json` (machine-readable).

Phase 5: Auto-Handoff

Trigger `/evals-implement` to generate code.

Verification

• Accepted drafts stored in `docs/adlc/memory/evals/EVAL-*.md`
• `evals/{system}/goldset.md` and `goldset.json` exist
• Holdout set `.adlc/memory/evals/holdout.json` isolated and populated
• All criteria are strictly binary (no confidence scores or Likert scales)
• Handover summary lists accepted criteria and adversarial counts