/validation-methodology

name: validation-methodology
description: Cross-cutting validation discipline. Acceptance criteria format, definition of done checklists, quality gate criteria, verification methods. Loaded by every analysis agent.

Validation Methodology

Specifications are claims. Claims require proof. This skill defines how the pipeline transforms claims into proof.

Acceptance Criteria Format

Every module spec receives formal acceptance criteria during Layer 4. Criteria use Given/When/Then format:

### AC-CONFIG-001: Config File Load
**Given** a valid config file exists at the documented path
**When** the application starts
**Then**
- The application reads every declared key from the file
- Unknown keys are logged as warnings, not errors
- Missing optional keys fall back to documented defaults

**Source:** official-docs, source-code, runtime-observation
**Confidence:** confirmed
**Priority:** P0
**Verification:** Automated test

ID Format: `AC-{DOMAIN}-{NNN}`

• `{DOMAIN}`: Short uppercase token for the behavioral domain (matches the spec's domain tokens — e.g., `CLI`, `CONFIG`, `API`, `AUDIO`, `PARSER`)
• `{NNN}`: Zero-padded three-digit sequence (001, 002, ...)
• IDs are immutable — retired, never reused

Field Rules

**Given/When/Then:**

• Use observable terms, not implementation details
• "Given a valid config file exists" not "Given the ConfigLoader is initialized"
• Each Then outcome must be verifiable without implementation internals
• Describe observable state, not UI mechanisms — "system rejects malformed input" not "error banner is visible." Criteria must be testable across any frontend platform.

**Priority:**

• P0: Must pass. Blocks pipeline. Implementation cannot ship without satisfying.
• P1: Should pass. Reported as warnings. Divergence requires documented rationale.
• P2: Nice to pass. Advisory.

**Verification:** One of the 5 methods below.

Output Location

workspace/raw/specs/validation/acceptance-criteria/
    _index.md                   # Summary with counts and links
    config-loading.md           # AC-CONFIG-001 through AC-CONFIG-NNN
    cli-flags.md                # AC-CLI-001 through ...
    ...

Verification Methods

Every acceptance criterion specifies how the implementer will verify it:

| Method | ID | When to Use | |--------|-----|------------| | **Automated Test** | `automated-test` | Precondition, action, and outcome are all programmable | | **Manual Inspection** | `manual-inspection` | Outcome requires human judgment (UX quality, clarity) | | **Runtime Observation** | `runtime-observation` | Behavior involves timing, async events, sustained observation | | **Protocol Capture** | `protocol-capture` | Wire format, headers, API protocol compliance | | **Code Review** | `code-review` | Architectural or quality criteria not directly testable at runtime |

Quality Gate Criteria

digraph gate_evaluation {
    rankdir=TB;

    "Agent produces output" [shape=ellipse];
    "Check DOD checklist for agent layer" [shape=box];
    "All DOD items pass?" [shape=diamond];
    "Proceed to gate evaluation" [shape=box];
    "Flag incomplete output for review" [shape=box];
    "Check gate criteria" [shape=box];
    "All gate criteria pass?" [shape=diamond];
    "Tag and proceed to next layer" [shape=box];
    "Remediate findings" [shape=box];
    "Attempts exhausted?" [shape=diamond];
    "STOP: Pipeline blocked" [shape=octagon, style=filled, fillcolor=red, fontcolor=white];

    "Agent produces output" -> "Check DOD checklist for agent layer";
    "Check DOD checklist for agent layer" -> "All DOD items pass?";
    "All DOD items pass?" -> "Proceed to gate evaluation" [label="yes"];
    "All DOD items pass?" -> "Flag incomplete output for review" [label="no"];
    "Flag incomplete output for review" -> "Proceed to gate evaluation";
    "Proceed to gate evaluation" -> "Check gate criteria";
    "Check gate criteria" -> "All gate criteria pass?";
    "All gate criteria pass?" -> "Tag and proceed to next layer" [label="yes"];
    "All gate criteria pass?" -> "Remediate findings" [label="no"];
    "Remediate findings" -> "Attempts exhausted?";
    "Attempts exhausted?" -> "STOP: Pipeline blocked" [label="yes"];
    "Attempts exhausted?" -> "Check gate criteria" [label="no"];
}

Gate 1 (spec-verifier, after Layer 3)

• [ ] Zero contradictions between specs
• [ ] All crypto claims and constants verified against source (not assumed)
• [ ] Every behavioral claim has a provenance citation
• [ ] Assumed claims are a small minority — most claims have direct evidence. A module dominated by assumptions needs more intelligence gathering before proceeding
• [ ] All modules have specs
• [ ] Zero uncited behavioral claims in output candidates

Gate 2 (spec-reviewer, after Layer 4)

• [ ] Zero implementation leakage in specs
• [ ] Zero P0 completeness gaps
• [ ] All ACs have IDs and are linked to specs
• [ ] P0 ACs have test vectors
• [ ] All ACs are testable
• [ ] All modules have ACs
• [ ] Gap report reviewed
• [ ] Zero contamination in output

Implementation Leakage Definition

"Zero implementation leakage" means every identifier, name, and reference in the output specs passes the reimplementor test: **"Could the reimplementor reasonably redesign this?"** If yes, it's an implementation detail that should have been abstracted. If no, it's an external contract that belongs in the spec.

**Implementation details (must not appear in output specs):**

| Category | What to look for | |----------|-----------------| | Internal names | Variable names, function signatures, class names, method names from the source — any language | | Internal architecture | Module boundaries, file organization, inheritance hierarchies described in prose | | Framework-specific patterns | State management internals, ORM patterns, framework lifecycle hooks by internal name | | Build/deployment artifacts | Source file paths, line numbers, chunk IDs, minifi