/evals-init

name: evals-init
description: Use when standing up evals/{system}/ for the first time — scaffolds the EDD directory structure, picks PromptFoo or DeepEval by tech stack, and generates a security baseline.
disable-model-invocation: true

evals-init

What this skill does

Initialize the **project-level evaluation directory structure** following EDD (Eval-Driven Development) principles to prepare for systematic evaluation development. This is completely standalone with zero spec-kit dependencies.

**Output**: 1. **Directory Structure** - `evals/{system}/` with proper organization (promptfoo | deepeval) 2. **Security Baseline** - Auto-created graders for PII leakage, prompt injection, hallucination detection, misinformation detection 3. **Configuration Files** - Standalone config.yml and goldset templates under `.adlc/evals/` 4. **Auto-handoff** to `/evals-specify` to begin error analysis

**Key EDD Principles Applied**:

• **Principle I**: Spec-Driven Contracts - Evals validate spec compliance
• **Principle II**: Binary Pass/Fail - No Likert scales in grader templates
• **Principle IV**: Evaluation Pyramid - Tier 1 (fast) + Tier 2 (goldset) structure
• **Principle IX**: Test Data as Code - Version control setup for datasets

When to use

• **Starting systematic evaluation**: Set up the initial evaluation harness for your application
• **EDD Adoption**: Converting from traditional testing to evaluation-driven development
• **Security-first evaluation**: Auto-generate baseline security checks from the start

When NOT to use

• **Evals directory already exists**: Use `/evals-validate` to run tests, or `/evals-specify` to add criteria
• **Evaluating team directives**: This is for project-level application behavior testing, not directives compliance

Process

User Input

$ARGUMENTS

Parse flags from the arguments first, then treat remaining text as focus areas:

• `--system SYSTEM` — Choose `promptfoo` or `deepeval`. If omitted, choose interactively based on tech stack.
• Remaining text — System description (focus setup)

Execution Steps

Phase 1: Tech Stack Detection

• Scan project manifests (`package.json`, `requirements.txt`, `Cargo.toml`, `go.mod`, etc.)
• Recommends PromptFoo for mixed/JS stacks; DeepEval for Python-native stacks

Phase 2: Create Directory Structure

Creates:

evals/
├── {system}/                    # promptfoo | deepeval
│   ├── goldset.md              # Published goldset
│   ├── goldset.json            # Auto-generated for system consumption
│   ├── config.yml              # System-specific configuration
│   ├── config.{js,py}          # Generated system config (.js for promptfoo, .py for deepeval)
│   └── graders/                # Binary pass/fail graders
│       ├── check_pii_leakage.py           # Security baseline
│       ├── check_prompt_injection.py     # Security baseline
│       ├── check_hallucination.py        # Security baseline
│       └── check_misinformation.py       # Security baseline
├── results/                    # Git-ignored run outputs
└── .adlc/
    └── drafts/evals/           # Draft eval records (Markdown + YAML)

Phase 3: Configuration Copy

• Create `.adlc/evals/` if missing.
• Copy `skills/evals/evals-templates/evals-config-template.yml` to `.adlc/evals/evals-config.yml`.

Phase 4: Auto-Handoff

Trigger `/evals-specify` to begin error analysis.

Verification

• `evals/{system}/goldset.md` exists (initially empty)
• `.adlc/evals/evals-config.yml` exists
• Graders directory populated with 4 security baseline python scripts
• Results directory contains `.gitignore` to prevent versioning traces
• Handover report generated with recommended framework and next steps